WebGUI - Why I hate the Syndicated Content asset today - Content Management System | CMS | Open Source Content Management | Web Application Framework

perlDreamer

Date: 8/29/2011 1:07 pm · Subject: Why I hate the Syndicated Content asset today · Rating: 0

Turns out that via Syndicated Content asset, you can put WebGUI into an infinite loop.

You put the Not Found Page on the site to the home page (default asset).

On that page, you put a Syndicated Content asset that refers to a page that does not exist on the site.

When the SC is rendered, then it tries to fetch that page which isn't found, which doesn't exist, so it gives you the Not Found Page, which is the home page, which tries to render the SC, and there's your loop.

If a site wants to DoS itself to death, it's not a problem until you get into shared hosting. Then you have a modperl process in runaway, which can quickly start affecting other sites as well as spectre.

The best solution I can think of is to not do real HTTP fetches on URLs that refer back to the original site. However, if you get someone determined to cause problems on a server, there are probably work-arounds.

Another solution would be not to have the Page Not Found page be an asset in the system, but rather an extended setting where you can supply a style template and some simple content.

Any other ideas, or rubuttals to what I've posted above?

Back to Top

Rate [ | ]

ehab

Date: 8/29/2011 2:30 pm · Subject: Re: Why I hate the Syndicated Content asset today · Rating: 0

Had problems with this to the point where i have stpped using it. Even small dns problems will bring a server to its knees.
--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

colink@perldreamer.com wrote:

perlDreamer wrote:

Turns out that via Syndicated Content asset, you can put WebGUI into an infinite loop.

You put the Not Found Page on the site to the home page (default asset).

On that page, you put a Syndicated Content asset that refers to a page that does not exist on the site.

When the SC is rendered, then it tries to fetch that page which isn't found, which doesn't exist, so it gives you the Not Found Page, which is the home page, which tries to render the SC, and there's your loop.

If a site wants to DoS itself to death, it's not a problem until you get into shared hosting. Then you have a modperl process in runaway, which can quickly start affecting other sites as well as spectre.

The best solution I can think of is to not do real HTTP fetches on URLs that refer back to the original site. However, if you get someone determined to cause problems on a server, there are probably work-arounds.

Another solution would be not to have the Page Not Found page be an asset in the system, but rather an extended setting where you can supply a style template and some simple content.

Any other ideas, or rubuttals to what I've posted above?

http://www.webgui.org/forums/dev/why-i-hate-the-syndicated-content-asset-today

--

WebGUI
http://www.webgui.org

Back to Top

Rate [ | ]

susanb

Date: 8/29/2011 3:41 pm · Subject: Re: Why I hate the Syndicated Content asset today · Rating: 0

Another solution would be not to have the Page Not Found page be an asset in the system, but rather an extended setting where you can supply a style template and some simple content.

If you decide to do this particular solution, I'd rather it were a dedicated but fully functional Page Layout type asset. A style template and some simple content, just sounds like it might be too limited in certain circumstances. Maybe someone wants to display different content dependant on the referrer of the bad page link, or some other things that I'm too uncreative to think of.

Susan B

Back to Top

Rate [ | ]

perlDreamer

Date: 8/29/2011 4:00 pm · Subject: Re: Why I hate the Syndicated Content asset today · Rating: 0

That would be done via macros, which could still be processed in the
content. You could even assetproxy in a Search form on the site.

Allowing an asset on that page is the cause of the problem in the first
place.

Back to Top

Rate [ | ]

preaction

Date: 8/29/2011 4:11 pm · Subject: Re: Why I hate the Syndicated Content asset today · Rating: 0

Frank and I had the idea to turn the Page Not Found page into just a template with some variables available to designers like the current URL.

While I was at it, I was going to make an Error template for more serious problems.

Back to Top

Rate [ | ]

ehab

Date: 8/29/2011 7:31 pm · Subject: Re: Why I hate the Syndicated Content asset today · Rating: 0

The page not found is not the only problem, a simple transient DNS problem will do the same. The solution is to make the code more resilient to arbitrary errors.

Ehab Heikal

Web design in Egypt, Arabic Portal with news chat and Videos using WebGUI

www.elmotaheda.com

Quote: An eye for an Eye only helps make the whole world blind

Gandhi

Back to Top

Rate [ | ]

preaction

Date: 8/29/2011 8:19 pm · Subject: Re: Why I hate the Syndicated Content asset today · Rating: 0

The live pull could be stopped, and only a cached version shown. Errors with pulling could be written to the logs, and those logs provided in an admin console plugin for site administrators. That would fix this and provide some helpful triage.

Back to Top

Rate [ | ]

rogier

Date: 8/30/2011 4:39 am · Subject: Re: Why I hate the Syndicated Content asset today · Rating: 0

...

Any other ideas, or rubuttals to what I've posted above?

We've had some problems with the SC asset. This homepage example is just one of the ways that the asset breaks: it also fails on malformed feeds and bad encoding. We use it on one site (7.8.16 stable) and it has on occasion broken the site. Removing the broken feed solves it. I don't think it has been fixed since.

To me it seems that the SC needs to be more robust generally. Too many feeds simply don't follow the RSS standards. Instead of only solving this problem, isn't there a generic way of testing whether the feed is present and then if it's valid (i.e. processable by the asset)?

Probably easier said than done. My knowledge on this topic is limited, just wanted to let you know there are other problems too.

Rogier | United Knowledge
www.unitedknowledge.nl · www.webgui-help.nl

Back to Top

Rate [ | ]

Trex

Date: 8/30/2011 9:42 am · Subject: Re: Why I hate the Syndicated Content asset today · Rating: 0

We've had some problems with the SC asset. This homepage example is just one of the ways that the asset breaks: it also fails on malformed feeds and bad encoding. We use it on one site (7.8.16 stable) and it has on occasion broken the site. Removing the broken feed solves it. I don't think it has been fixed since.

To me it seems that the SC needs to be more robust generally. Too many feeds simply don't follow the RSS standards. Instead of only solving this problem, isn't there a generic way of testing whether the feed is present and then if it's valid (i.e. processable by the asset)?

I have also found that the SC asset has room for improvement in terms of robustness and error handling. One of the sites I manage pulled RSS data from a LiveJournal blog -- until the recent DDoS attack on LiveJournal. The LiveJournal outage's effect on our site via the SC feed didn't take down our site, but it did impact performance of the site in general. (Unfortunately, I was quick to remove the SC feed and don't recall specifics about its impact on the site in general.)

However, I do remember that it did have a major impact on administrators: rendering of a page with the LJ feed in admin-on mode rolled to a stop and hung for what seemed to be several minutes until something timed out and the page finally finished rendering. It was as if WebGUI REALLY wanted to get the latest content for admins, and it would wait all day until it got it if the RSS feed showed any sign of life and wasn't completely dead (as was the case with this DDoS attack).

One thing that would be an improvement to the SC asset would be some sort of archiving of RSS feeds. This would be different from caching, because the archive would hang around indefinitely: caching could be set to a low number to keep your SC synchronized with the RSS feed, but the archive would always be available if the RSS feed becomes unavailable.

If such an archive were available, you'd also want a template variable that indicates when the archived content was last updated, so that you could provide information to the viewers that the source of the content is down, but here's some archived content as of {some date}. Site admins could watch the dates to determine when a feed is no longer suitable for their site.

In terms of archive storage, the simplest solution would be to store the latest RSS response as the whole XML document. However, you could also break the RSS items off into separate collaboration-system-like posts. With that, you could actually use the SC as a long-term archive of a particular feed, saving the old posts as they get bumpped off the bottom of the original feed. This might not be useful for every SC (and might run afoul of some RSS feed usage rules), so you'd want it to be an option configurable for each SC.

Back to Top

Rate [ | ]

arjan

Date: 8/30/2011 10:05 am · Subject: Re: Why I hate the Syndicated Content asset today · Rating: 0

Another example of a problem may be:

try setting the permissions of a feed you want to pull with the Syndicated Content asset to admins only and place a dozen SC's on your site. They will all wait for a full minute and consume all your modperl-processes, leaving nothing for your real visitors. I suggest this activity should always run in serial.

Kind regards,
Arjan.

On 08/30/2011 04:42 PM, daletrexel@gmail.com wrote:

Trex wrote:

We've had some problems with the SC asset. This homepage example is just one of the ways that the asset breaks: it also fails on malformed feeds and bad encoding. We use it on one site (7.8.16 stable) and it has on occasion broken the site. Removing the broken feed solves it. I don't think it has been fixed since.

To me it seems that the SC needs to be more robust generally. Too many feeds simply don't follow the RSS standards. Instead of only solving this problem, isn't there a generic way of testing whether the feed is present and then if it's valid (i.e. processable by the asset)?

I have also found that the SC asset has room for improvement in terms of robustness and error handling. One of the sites I manage pulled RSS data from a LiveJournal blog -- until the recent DDoS attack on LiveJournal. The LiveJournal outage's effect on our site via the SC feed didn't take down our site, but it did impact performance of the site in general. (Unfortunately, I was quick to remove the SC feed and don't recall specifics about its impact on the site in general.)

However, I do remember that it did have a major impact on administrators: rendering of a page with the LJ feed in admin-on mode rolled to a stop and hung for what seemed to be several minutes until something timed out and the page finally finished rendering. It was as if WebGUI REALLY wanted to get the latest content for admins, and it would wait all day until it got it if the RSS feed showed any sign of life and wasn't completely dead (as was the case with this DDoS attack).

One thing that would be an improvement to the SC asset would be some sort of archiving of RSS feeds. This would be different from caching, because the archive would hang around indefinitely: caching could be set to a low number to keep your SC synchronized with the RSS feed, but the archive would always be available if the RSS feed becomes unavailable.

If such an archive were available, you'd also want a template variable that indicates when the archived content was last updated, so that you could provide information to the viewers that the source of the content is down, but here's some archived content as of {some date}. Site admins could watch the dates to determine when a feed is no longer suitable for their site.

In terms of archive storage, the simplest solution would be to store the latest RSS response as the whole XML document. However, you could also break the RSS items off into separate collaboration-system-like posts. With that, you could actually use the SC as a long-term archive of a particular feed, saving the old posts as they get bumpped off the bottom of the original feed. This might not be useful for every SC (and might run afoul of some RSS feed usage rules), so you'd want it to be an option configurable for each SC.

http://www.webgui.org/forums/dev/why-i-hate-the-syndicated-content-asset-today/8

--

WebGUI
http://www.webgui.org

-- 
NIEUW: http://www.mediacalculator.unitedknowledge.nl/

Hoe verslaan de media het politieke nieuws? Wie haalt het nieuws en hoe werkt dat uit? Bekijk het in de MediaCalculator: mediacalculator.unitedknowledge.nl

Recent: http://www.lomcongres.nl/
Congres- en nieuwsbriefportaal met relatiebeheer systeem voor het Landelijk Overleg Milieuhandhaving

Setting Standards, a Delft University of Technology and United Knowledge simulation exercise on strategy and cooperation in standardization, http://www.setting-standards.com

United Knowledge, internet voor de publieke sector
Keizersgracht 74
1015 CT Amsterdam
T +31 (0)20 52 18 300
F +31 (0)20 52 18 301
bureau@unitedknowledge.nl
http://www.unitedknowledge.nl

M +31 (0)6 2427 1444
E arjan@unitedknowledge.nl

Bezoek onze site op:
http://www.unitedknowledge.nl

Of bekijk een van onze projecten:
http://www.handhavingsportaal.nl/
http://www.setting-standards.com/
http://www.lomcongres.nl/
http://www.clubvanmaarssen.org/

Back to Top

Rate [ | ]

Previous · Next
User	Message
perlDreamer	Date: 8/29/2011 1:07 pm · Subject: Why I hate the Syndicated Content asset today · Rating: 0 Turns out that via Syndicated Content asset, you can put WebGUI into an infinite loop. You put the Not Found Page on the site to the home page (default asset). On that page, you put a Syndicated Content asset that refers to a page that does not exist on the site. When the SC is rendered, then it tries to fetch that page which isn't found, which doesn't exist, so it gives you the Not Found Page, which is the home page, which tries to render the SC, and there's your loop. If a site wants to DoS itself to death, it's not a problem until you get into shared hosting. Then you have a modperl process in runaway, which can quickly start affecting other sites as well as spectre. The best solution I can think of is to not do real HTTP fetches on URLs that refer back to the original site. However, if you get someone determined to cause problems on a server, there are probably work-arounds. Another solution would be not to have the Page Not Found page be an asset in the system, but rather an extended setting where you can supply a style template and some simple content. Any other ideas, or rubuttals to what I've posted above?
Back to Top	Rate [ \| ]

ehab	Date: 8/29/2011 2:30 pm · Subject: Re: Why I hate the Syndicated Content asset today · Rating: 0 Had problems with this to the point where i have stpped using it. Even small dns problems will bring a server to its knees. -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. colink@perldreamer.com wrote: perlDreamer wrote: Turns out that via Syndicated Content asset, you can put WebGUI into an infinite loop. You put the Not Found Page on the site to the home page (default asset). On that page, you put a Syndicated Content asset that refers to a page that does not exist on the site. When the SC is rendered, then it tries to fetch that page which isn't found, which doesn't exist, so it gives you the Not Found Page, which is the home page, which tries to render the SC, and there's your loop. If a site wants to DoS itself to death, it's not a problem until you get into shared hosting. Then you have a modperl process in runaway, which can quickly start affecting other sites as well as spectre. The best solution I can think of is to not do real HTTP fetches on URLs that refer back to the original site. However, if you get someone determined to cause problems on a server, there are probably work-arounds. Another solution would be not to have the Page Not Found page be an asset in the system, but rather an extended setting where you can supply a style template and some simple content. Any other ideas, or rubuttals to what I've posted above? http://www.webgui.org/forums/dev/why-i-hate-the-syndicated-content-asset-today -- WebGUI http://www.webgui.org
Back to Top	Rate [ \| ]

susanb	Date: 8/29/2011 3:41 pm · Subject: Re: Why I hate the Syndicated Content asset today · Rating: 0 Another solution would be not to have the Page Not Found page be an asset in the system, but rather an extended setting where you can supply a style template and some simple content. If you decide to do this particular solution, I'd rather it were a dedicated but fully functional Page Layout type asset. A style template and some simple content, just sounds like it might be too limited in certain circumstances. Maybe someone wants to display different content dependant on the referrer of the bad page link, or some other things that I'm too uncreative to think of. Susan B
Back to Top	Rate [ \| ]

perlDreamer	Date: 8/29/2011 4:00 pm · Subject: Re: Why I hate the Syndicated Content asset today · Rating: 0 That would be done via macros, which could still be processed in the content. You could even assetproxy in a Search form on the site. Allowing an asset on that page is the cause of the problem in the first place.
Back to Top	Rate [ \| ]

preaction	Date: 8/29/2011 4:11 pm · Subject: Re: Why I hate the Syndicated Content asset today · Rating: 0 Frank and I had the idea to turn the Page Not Found page into just a template with some variables available to designers like the current URL. While I was at it, I was going to make an Error template for more serious problems.
Back to Top	Rate [ \| ]

ehab	Date: 8/29/2011 7:31 pm · Subject: Re: Why I hate the Syndicated Content asset today · Rating: 0 The page not found is not the only problem, a simple transient DNS problem will do the same. The solution is to make the code more resilient to arbitrary errors. Ehab Heikal Web design in Egypt, Arabic Portal with news chat and Videos using WebGUI www.elmotaheda.com Quote: An eye for an Eye only helps make the whole world blind Gandhi
Back to Top	Rate [ \| ]

preaction	Date: 8/29/2011 8:19 pm · Subject: Re: Why I hate the Syndicated Content asset today · Rating: 0 The live pull could be stopped, and only a cached version shown. Errors with pulling could be written to the logs, and those logs provided in an admin console plugin for site administrators. That would fix this and provide some helpful triage.
Back to Top	Rate [ \| ]

rogier	Date: 8/30/2011 4:39 am · Subject: Re: Why I hate the Syndicated Content asset today · Rating: 0 ... Any other ideas, or rubuttals to what I've posted above? We've had some problems with the SC asset. This homepage example is just one of the ways that the asset breaks: it also fails on malformed feeds and bad encoding. We use it on one site (7.8.16 stable) and it has on occasion broken the site. Removing the broken feed solves it. I don't think it has been fixed since. To me it seems that the SC needs to be more robust generally. Too many feeds simply don't follow the RSS standards. Instead of only solving this problem, isn't there a generic way of testing whether the feed is present and then if it's valid (i.e. processable by the asset)? Probably easier said than done. My knowledge on this topic is limited, just wanted to let you know there are other problems too. Rogier \| United Knowledge www.unitedknowledge.nl · www.webgui-help.nl
Back to Top	Rate [ \| ]

Trex	Date: 8/30/2011 9:42 am · Subject: Re: Why I hate the Syndicated Content asset today · Rating: 0 We've had some problems with the SC asset. This homepage example is just one of the ways that the asset breaks: it also fails on malformed feeds and bad encoding. We use it on one site (7.8.16 stable) and it has on occasion broken the site. Removing the broken feed solves it. I don't think it has been fixed since. To me it seems that the SC needs to be more robust generally. Too many feeds simply don't follow the RSS standards. Instead of only solving this problem, isn't there a generic way of testing whether the feed is present and then if it's valid (i.e. processable by the asset)? I have also found that the SC asset has room for improvement in terms of robustness and error handling. One of the sites I manage pulled RSS data from a LiveJournal blog -- until the recent DDoS attack on LiveJournal. The LiveJournal outage's effect on our site via the SC feed didn't take down our site, but it did impact performance of the site in general. (Unfortunately, I was quick to remove the SC feed and don't recall specifics about its impact on the site in general.) However, I do remember that it did have a major impact on administrators: rendering of a page with the LJ feed in admin-on mode rolled to a stop and hung for what seemed to be several minutes until something timed out and the page finally finished rendering. It was as if WebGUI REALLY wanted to get the latest content for admins, and it would wait all day until it got it if the RSS feed showed any sign of life and wasn't completely dead (as was the case with this DDoS attack). One thing that would be an improvement to the SC asset would be some sort of archiving of RSS feeds. This would be different from caching, because the archive would hang around indefinitely: caching could be set to a low number to keep your SC synchronized with the RSS feed, but the archive would always be available if the RSS feed becomes unavailable. If such an archive were available, you'd also want a template variable that indicates when the archived content was last updated, so that you could provide information to the viewers that the source of the content is down, but here's some archived content as of {some date}. Site admins could watch the dates to determine when a feed is no longer suitable for their site. In terms of archive storage, the simplest solution would be to store the latest RSS response as the whole XML document. However, you could also break the RSS items off into separate collaboration-system-like posts. With that, you could actually use the SC as a long-term archive of a particular feed, saving the old posts as they get bumpped off the bottom of the original feed. This might not be useful for every SC (and might run afoul of some RSS feed usage rules), so you'd want it to be an option configurable for each SC.
Back to Top	Rate [ \| ]

arjan	Date: 8/30/2011 10:05 am · Subject: Re: Why I hate the Syndicated Content asset today · Rating: 0 Another example of a problem may be: try setting the permissions of a feed you want to pull with the Syndicated Content asset to admins only and place a dozen SC's on your site. They will all wait for a full minute and consume all your modperl-processes, leaving nothing for your real visitors. I suggest this activity should always run in serial. Kind regards, Arjan. On 08/30/2011 04:42 PM, daletrexel@gmail.com wrote: Trex wrote: We've had some problems with the SC asset. This homepage example is just one of the ways that the asset breaks: it also fails on malformed feeds and bad encoding. We use it on one site (7.8.16 stable) and it has on occasion broken the site. Removing the broken feed solves it. I don't think it has been fixed since. To me it seems that the SC needs to be more robust generally. Too many feeds simply don't follow the RSS standards. Instead of only solving this problem, isn't there a generic way of testing whether the feed is present and then if it's valid (i.e. processable by the asset)? I have also found that the SC asset has room for improvement in terms of robustness and error handling. One of the sites I manage pulled RSS data from a LiveJournal blog -- until the recent DDoS attack on LiveJournal. The LiveJournal outage's effect on our site via the SC feed didn't take down our site, but it did impact performance of the site in general. (Unfortunately, I was quick to remove the SC feed and don't recall specifics about its impact on the site in general.) However, I do remember that it did have a major impact on administrators: rendering of a page with the LJ feed in admin-on mode rolled to a stop and hung for what seemed to be several minutes until something timed out and the page finally finished rendering. It was as if WebGUI REALLY wanted to get the latest content for admins, and it would wait all day until it got it if the RSS feed showed any sign of life and wasn't completely dead (as was the case with this DDoS attack). One thing that would be an improvement to the SC asset would be some sort of archiving of RSS feeds. This would be different from caching, because the archive would hang around indefinitely: caching could be set to a low number to keep your SC synchronized with the RSS feed, but the archive would always be available if the RSS feed becomes unavailable. If such an archive were available, you'd also want a template variable that indicates when the archived content was last updated, so that you could provide information to the viewers that the source of the content is down, but here's some archived content as of {some date}. Site admins could watch the dates to determine when a feed is no longer suitable for their site. In terms of archive storage, the simplest solution would be to store the latest RSS response as the whole XML document. However, you could also break the RSS items off into separate collaboration-system-like posts. With that, you could actually use the SC as a long-term archive of a particular feed, saving the old posts as they get bumpped off the bottom of the original feed. This might not be useful for every SC (and might run afoul of some RSS feed usage rules), so you'd want it to be an option configurable for each SC. http://www.webgui.org/forums/dev/why-i-hate-the-syndicated-content-asset-today/8 -- WebGUI http://www.webgui.org -- NIEUW: http://www.mediacalculator.unitedknowledge.nl/ Hoe verslaan de media het politieke nieuws? Wie haalt het nieuws en hoe werkt dat uit? Bekijk het in de MediaCalculator: mediacalculator.unitedknowledge.nl Recent: http://www.lomcongres.nl/ Congres- en nieuwsbriefportaal met relatiebeheer systeem voor het Landelijk Overleg Milieuhandhaving Setting Standards, a Delft University of Technology and United Knowledge simulation exercise on strategy and cooperation in standardization, http://www.setting-standards.com United Knowledge, internet voor de publieke sector Keizersgracht 74 1015 CT Amsterdam T +31 (0)20 52 18 300 F +31 (0)20 52 18 301 bureau@unitedknowledge.nl http://www.unitedknowledge.nl M +31 (0)6 2427 1444 E arjan@unitedknowledge.nl Bezoek onze site op: http://www.unitedknowledge.nl Of bekijk een van onze projecten: http://www.handhavingsportaal.nl/ http://www.setting-standards.com/ http://www.lomcongres.nl/ http://www.clubvanmaarssen.org/
Back to Top	Rate [ \| ]