Roxen WebServer 5.0.449-release3 (2010-02-08)

WebServer-specific changes

Administration Interface improvements:

  • Site: Reduce SNMP OIDs clutter.
  • Site: Fixed backtrace that could happen when adding a port to a site. [Bug 5047]
  • Tasks: Debug info page now includes environment2 in output.
  • Tasks: Debug info page should be more tolerant to exceptions in _sprintf().
  • Tasks: Hide/show toggles in backtrace wizard.
  • Tasks: Make OK button on patch page link back to list.
  • DB: Fix for problem where "" is shown as 0.
  • DB: Don't run out of memory when importing db dump. [Bug 5190]
  • DB: Database backup now shows $VARDIR path in dialog.

Replication improvements:

  • Make sure database alias points to a database. [Bug 4698]
  • Improve warning for unconfigured database.

RXML improvements:

  • Fix missing variable type conversion for entities with encodings.
  • New flag that forces tag content to be valid value instead of RXML.nil.
  • Added timeout parameters to <gtext>, <cimg>, <gbutton>, <atlas>, <gxml> and <diagram>.
  • Fixed filename attribute in <cimg-url>.
  • Support internal images in <gxml>. [Bug 4888]
  • Correct <gxml> format strings when using coordinate systems. [Bug 4924]
  • Include <gxml> documentation.
  • Quote wide characters in JavaScript support module. [Bug 4904]
  • Fixed quoting issue in <insert#cached-href>.
  • Corrected use of timezone in <date http-time="..."/>.
  • Fix bug in s command in <sed>.
  • Fix scope cloning when evaluating p-code.
  • Fix <sort> with empty content. [Bug 5219]
  • Allow use of remote database in <accessed/>.

Core improvements:

  • New start()/stop() callbacks and tightening of bootstrap_info.
  • Allow zero resource in log_event().
  • register_vary_callback() promotes 0 to *.
  • Protect against zone_to_ascii()throwing errors.
  • Robustness fixes in reading of configuration.
  • Improved robustness to strange input in find_ips_for().
  • Only set LD_ASSUME_KERNEL if supported.
  • Roxen.SqlNull now based on Sql.Null.
  • Ignore port number in Preferred Language filter Match host name. [Bug 4882]
  • Improve error message for proxy connection failures.
  • Protect pike script instances during execution. [Bug 5072]
  • Fixed various bugs in HTTP layer.
  • Avoid loading modules several times when instantiating a new site from template.
  • Added old-password auth method compatible with MySQL 4.0.
  • MySQL logs to console when running with --once.
  • Remove unwanted inherit from patcher.pike.
  • FastCGI race-condition and infinite loop fixes. [Bug 5213]
  • Improve debug when wide strings are written to access log.
  • Many other fixes and changes.



Pike-specific changes

Module improvements:

  • Calendar: Updated timezone data.
  • LDAP: More tolerant to various invalid characters that occur in reality.
  • LDAP: Fixed noise about root DSE attributes without schema.
  • SSL: Disable renegotiation to avoid man-in-the-middle attack.
  • SSL: Don't hog the backend thread if network is fast enough to keep up with encryption.
  • SSL and Stdio: Significant performance improvements when sending large files.
  • MySQL: Fixed initialization of charset after reconnect.
  • MySQL: Implemented big_typed_query() and streaming_typed_query().
  • MySQL: Fix infinite loop for unbalanced quotes.
  • SQL: Added Sql.Null and fetch_json_result().
  • Standards.URI: Better handling of empty URIs and error handling in query variable access.
  • Web.Crawler: Avoid getting stuck on errors.
  • Gz: Fixed stray lock if error occurs during inflate or deflate.
  • Image: Fix various memory issues in PNG, WBF and XWD decoders.
  • Image: decode_header() support for TIFF and PS.
  • Graphics.Graph: Fixed issues with axis labels and scale.
  • Protocols.DNS: Support for async replies and multiple port/interface combinations.
  • String: Added String.normalize_space().

Core improvements:

  • Speed up string searches for single-character strings.
  • Speed up appending a single element to an array.
  • Workaround for bug in Windows with paths starting with a wide character.
  • Fix buffer overflow for float to string conversion.
  • Fix regression from Pike 7.4 in integer/floating-point comparisons.
  • Fix detection of POSIX high-resolution timers.
  • Update Mac OS X/x86 compilation flags to use SSE math.
  • Call compile_warning() in current master.
  • Improve line number reporting in error messages.
  • Various internal fixes. [Bug 5086] [Bug 2672] [Bug 4351] [Bug 5085] [Bug 4257] [Bug 2823]
  • Many other fixes and changes.



MySQL-specific changes

Updated the bundled MySQL to version 5.0.90 which includes many stability and performance fixes. To see the changes from the previous 5.0.83 package please read the following MySQL change notes:

NOTE: Roxen distributions for Mac OS X 10.4 (PowerPC 32-bit) include MySQL 5.0.87 since MySQL no longer releases PowerPC binaries.


Roxen WebServer 5.0.403-release2 (2009-07-08)

WebServer-specific changes

RXML improvements:

  • Fixed bug that made &page.ssl-strength; return incorrect data.
  • Corrected a problem that could cause Bad argument 1 to finish(). Expected string. error messages.
  • Made the RXML parser more strict about typing of tags that are listed as required or optional attributes.

Databases improvements:

  • Improved syntax parsing for the MySQL 5.0 migration script.
  • Implemented --mysql-log-queries option to the start script to get detailed logging of all MySQL queries.

Administration Interface improvements:

  • Clicks on Restore default value buttons in the administration interface now require confirmation before resetting the value.
  • Relaxed the authentication requirements for dynamically generated images in the Administration interface to reduce the number of dialog boxes that appear when logging in.

Core improvements:

  • Fixed memory garbage caused by calling internal scheduling function (roxen.background_run()).
  • Added log options $queue-time, $queue-length, $handle-time and $handle-cputime.
  • Fixed bug in Roxen patch command rxnpatch when listing changed files.
  • Now correctly supports FTP clients that close the command channel before the data transfer is complete.
  • Fixed a IPv6-related problem where queries for the current configuration's IP address could return [::] (ANY) instead of [::1] (localhost).



Pike-specific changes

Modules improvements:

  • Fixed multiple issues (Content-Length header, callback timeouts, SSL handling) in Protocols.HTTP.Query.

Core improvements:

  • Corrected a problem where (string) 1.0 lost the fractional part.
  • Fixed a problem that made calls to sendfile() block when they should be running asynchronously.
  • Turned off obsolete FILE_STAT_CACHE mode internally in the master.
  • Fixed inheritance bug related to identifier visibility.
  • Improved display of backtraces.



MySQL-specific changes

Updated the bundled MySQL to version 5.0.83 which includes a large number of stability and performance fixes. To see the changes from the previous 5.0.77 package please read the following MySQL change notes:


Roxen WebServer 5.0.381-release1 (2009-05-29)

Note: This is a major release, and as such there are more structural changes than usual which have larger effects on compatibility. Wherever feasible, compatibility is kept through the compatibility level setting (found under the Settings tab for each site). There are however several incompatibilities that are not overcome that way. All compatibility and upgrade considerations are detailed in the sections labelled COMPAT NOTE below.

IMPORTANT: Note in particular that when you upgrade an existing Roxen installation, you cannot downgrade again in a simple way, so please make sure you have a backup of the configurations directory before you install Roxen 5.0. The reason is the update of the MySQL server to 5.0, which causes an automatic migration of the MySQL databases.

WebServer-specific changes

Core improvements:

  • Moved to Pike version 7.8, which among many other things enables full use of 64-bit hardware on most OS:es. Roxen 5.0 does not run on any earlier Pike version.

    COMPAT NOTE: There are a number of incompatibilities between Pike 7.4 and 7.8, but that is of no concern if you do not have your own custom modules. Roxen module developers should take a look at the Pike release notes for 7.6 and 7.8. Most incompatibilities are mitigated by running in 7.4 compatibility mode, which is enabled by simply putting "#pike 7.4" at the top of each pike (and pmod) file.

  • Moved to MySQL version 5.0. Roxen no longer works with an earlier version. By default it does not accept 5.1 or any later version either, since that combination is unsupported. That check can however be disabled with a define ALLOW_UNSUPPORTED_MYSQL.

    COMPAT NOTE: There are many MySQL compatibility considerations, but unless you have your own databases or tables in the Roxen MySQL you need not bother very much with them. Otherwise, please consult the upgrade notes in the MySQL manual for further details, for instance to get information on how to update your query syntax. Note that there might be actions you need to do on your MySQL data before upgrading.

    On upgrade, Roxen automatically updates the privilege tables in MySQL to cover all new privileges, similar to what the mysql_upgrade script does. Read-only access in the DB interface now translates to Select_priv and Execute_priv, and read/write access gives Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv, Drop_priv, References_priv, Index_priv, Alter_priv, Create_tmp_table_priv, Lock_tables_priv, Create_view_priv, Show_view_priv, Create_routine_priv, Alter_routine_priv, and Execute_priv.

  • It is now easier to make Roxen use another MySQL installation, either one separately installed or one that comes with the OS distribution. Paths to the MySQL installation directory and the most important executables are specified in the file mysql-location.txt in the Roxen server directory.

  • Databases: Introduced charset support for database connections. The server can now keep track of the charset a database uses for queries and returned text, and automatically do conversions to the Unicode representation used internally in RXML etc. This currently works for MySQL and Postgres (using the new pgsql driver). (Later 4.5 releases also had this feature to a limited degree.)

  • Databases: Fixed security issues where arbitrary databases in the Roxen server could be reached through the SQL tags. The SQL tags module now has a setting Allowed databases that specifies which databases may be accessed through the SQL tags. This setting is also used by some other modules that access databases, e.g. <emit source="timerange" query="...">.

    COMPAT NOTE: Database access is disabled by default, meaning that essentially all SQL tags that don't use the default database will not work until the Allowed databases setting has been configured to local needs. The module logs connection attempts to blocked databases in the event log, so it is possible to see that way which ones that need to be let through.

  • Added new start up script for Roxen CMS on OS X. Handles start, stop and restart.

  • IPv6 support. Roxen can now bind IPv6 ports, initiate connections to other IPv6 servers, etcetera. (4.5 also had this to a limited degree, if run on Pike 7.6 instead of the included 7.4.)

  • SNMP (Simple Network Management Protocol) support. This allows monitoring of the Roxen server through SNMP v1 or v2c (but not v3). It is enabled simply by registrering another port with protocol snmp for a site configuration. Each site has its own SNMP prefix, so several sites can share the same SNMP port.

  • Removed the old argcache system so that the new one is used by default (i.e. ENABLE_NEW_ARGCACHE is no longer necessary). This primarily means that links to auto-generated images are handled in a more robust way, especially in replicated setups.

    COMPAT NOTE: There is no compatibility fallback to read argcache entries from the old database. An upgrade instead depends on that the image cache remains intact for long enough so that old images can be served without requiring their argcache entries.

    COMPAT NOTE: The old arguments table in the local database is not dropped automatically. Administrators are adviced to do that to free up space.

  • The server now automatically runs a consistency check of the MySQL tables at startup.

  • Experimental support for gzip compression of http responses. Enabled by the define HTTP_COMPRESSION.

  • The protocol cache can now continue to deliver stale cache entries while new results are generated in the background. This can mitigate very long response times and server choking when a cache further back is being invalidated (typically the persistent disk cache in the CMS). (This was also available as an experimental feature in 4.5rel4 when ENABLE_SPCI was defined.)

  • Handling of incoming and outgoing charsets has been improved and primarily works better in UTF-8 mode. UTF-8 is also the default charset for form and query variables, with fallback to ISO-8859-1 if UTF-8 decode fails. That should lessen the need for <roxen-automatic-charset-variable> in forms (although it still is useful to make forms work better for users with really old browsers).

  • The size calculations for entries in the various RAM caches (seen under Tasks/Status/Cache status in the Administration Interface) are now much more accurate.

  • Added support for software packages. This is a system to be able to install complex module distributions without putting everything in the Roxen module path.

  • COMPAT NOTE: Roxen is no longer distributed with its own copies of fonts that may override bitmap versions shipped to customers, since the FreeType renderings are different enough to affect spacing etc.

  • COMPAT NOTE: If a port is opened for more than one server configuration, the fallback in case there is no site with the "Default site" flag set in the server has changed slightly: Now the configuration with the least specific port URL is used, while in earlier versions the most specific port URL was chosen. This is a fringe case that should only affect badly configured servers.

Administration interface improvements:

  • New patch management system, usable both to apply patches from Roxen Internet Software and to manage your own local changes. It is accessible from Tasks/Maintenance/Patch management, and there is also a command line tool bin/rxnpatch in the server directory.

  • The database browser under the DBs tab has been overhauled and is now much faster and has more features. Among other things there are tools to optimize or repair tables, and the permission matrix is on a separate subtab.

  • The Tasks/Debug Information/Resolve Path dialogue can now send form variables and cookies to simulate sessions and to aid in debugging input forms. The logging has also been extended to provide more details from various facilities.

  • Added a debug tool that dumps all threads whenever any handler thread or background job has been running for a set number of seconds. It's configurable from Globals/Logging and can be disabled with the define NO_SLOW_REQ_BT.

  • Support IDN hostnames for port bindings.

  • Added experimental support for scheduled database backups. It is only enabled if ENABLE_DB_BACKUPS is defined, and it then adds a new tag DBs/Backup schedules.

RXML core improvements:

  • The RXML type system has been extended with array and mapping types, and it is now fully deployed in various RXML tags through type attributes and type context sensitivity. See the new RXML Type System chapter in the Web Developer manual for all the details.

  • Cleaned up handling of the RXML nil value (RXML.nil):

    1. Do not allow an RXML variable to be set to RXML.nil. That deletes the variable instead.
    2. <emit source="sql" ...> maps SQL NULL to a null value instead of RXML.nil.
    3. <if variable="var.foo"> is false both for undefined variables (i.e. RXML.nil) and the SQL null values.
    4. A new test <if variable-exists="var.foo"> is added to test whether a variable is defined or not, i.e. it is true for null values but not for undefined variables.
    5. <if sizeof=...> evaluates to zero both for undefined and null variables, for compatibility.
    6. <copy-scope> no longer copies undefined variables.
    7. <emit source="values"> and <insert> sources variables and scopes no longer list undefined variables.

    COMPAT NOTE: Items 2, 6 and 7 are only activated if the compat level is 5.0 or higher (items 3 and 5 don't affect compatibility since null values didn't exist earlier).

  • Added an <emit> attribute filter-exclude which does the opposite of filter.

  • Added encodings utf16, utf16be, utf16le and hex which can be used in the encoding spec for variable entities, e.g. &var.x:hex;.

  • Accessing the cookie scope no longer implicitly disables the protocol cache. The protocol cache instead starts to vary on the cookie value.

    COMPAT NOTE: The old behavior meant that any RXML accessing cookies implicitly disabled the protocol cache. That means the new behavior can introduce overcaching side-effects in old code, even if the protocol cache entries are cookie specific. Therefore the cache-disabling behavior is kept if the compat level is 4.5 or earlier.

  • The truth value (page.last-true) is now always set to false when an RXML error is caught.

    COMPAT NOTE: This is a change that might have compat implications. It's only enabled on compat level 5.0.

  • The formatting of the online tag documentation has been improved and shows the structure more clearly.

RXML tag improvements:

  • Added a <value> tag to do type casting and to build compound values like arrays and mappings.

  • Added a <substring> tag that can pick out parts of strings in a number of ways.

  • Added a <range> tag which is similar to <substring> but operates on arrays instead.

  • <insert source="variables"> is made context sensitive so that it returns the scope mapping as-is in an array or mapping context.

  • The type handling in the <set> and <append> tags has been cleaned up.

    COMPAT NOTE: These changes have compatibility effects in some cases. Their old behavior is retained on the 4.5 compat level.

    In particular, the <append> tag behaves differently with arrays now. If you get errors in code that appends string elements to arrays, then you probably want to add type="text/*" to the <append> tag.

  • Extended the <set expr=...> expressions to allow various set operations on arrays and mappings. The docs for it is also adequate now.

  • SQL tags: The charset handling has been changed to fit the design principle that strings are always unencoded Unicode internally in RXML. It therefore not only controls the connection charset, but perhaps more importantly it also encodes queries and decodes results with that charset. That makes it useful also with databases without charset support in the client library.

    COMPAT NOTE: This change of the charset handling is incompatible. The old behavior is retained on the 4.5 compat level.

  • Additional RXML tags: Added two tags <dirname> and <basename> to pick out the respective parts of a path.

  • Additional RXML tags: Added an <xml-rpc-call> tag to make simple synchronous xml-rpc calls. It is enabled by the same option that enables synchronous <insert href>.

  • Additional RXML tags: Made it possible to pass data content in <insert href> POST requests.

  • Image converter and GXML modules: A filename attribute has been added to the <cimg> and GXML tags to append a filename to the autogenerated URL. The GXML module has also been blessed with the option to add image extensions to the URLs, like <cimg> already could do.

  • Graphics tags: Several of the attributes that these tags accept refer to files that are used for purposes such as textures or backgrounds. We now properly register timestamps to detect changes for some attributes that weren't handled earlier. If any of these resources are protected the RXML parser will now answer with a HTTP Auth Required response to the browser to force authentication instead of generating and caching an incorrect image.

  • Business graphics: Introduced a color-scheme attribute to the <diagram> tag to autogenerate colors for data.

  • E-mail module: Improved error handling in the <email> tag and added a new attribute error-variable. Also added an attribute envelope-from to set the envelope sender address.

  • Added type attribute to <redirect> to make it possible to do e.g. permanent redirects.

  • Added attribute http-time to <date>.

  • Added new mode safe-utf8 to <recode from>. When provided, silently ignore any illegal UTF-8 sequences.

  • Added a showvar attribute to the <debug> tag to be able to print out the value of a variable without conversion in an unambiguous format (useful to figure out charset conversion issues, for instance). The <debug> tag also works in any type context.

  • COMPAT NOTE: A bug has been fixed in the <contents> tag, used within <define>, when both the value-of and result-set attributes were used at the same time. The fix might have compatibility implications in code that tried to work around the bug, but the old behavior was too quirky to keep compatibility with on the 4.5 compat level.

  • COMPAT NOTE: A type problem that could cause extra entity quoting in <strlen> has been fixed and hence wrong length reports. Old code might possibly depend on the broken behavior, so it is kept on 4.5 compat level.

Module improvements:

  • The Yahoo! User Interface Library: This is a new module can be used to make the YUI available to web applications on the site. It supports several YUI versions simultaneously and it is easy to add another by downloading the tar distribution and letting the module unpack it.

  • Path info support: Added path limit option.

  • Redirect module: Added permanent keyword to send permanent (301) redirects instead of temporary (302).

  • RXML parser module: Added an option to censor potentially sensitive user authentication data from requests before RXML evaluation starts.

  • CGI interface now exports REQUEST_URI, REDIRECT_URL and REDIRECT_STATUS.

  • CGI scripting support: Added a setting to be able to run CGI scripts in a chrooted environment.

  • CGI scripting support: Added a setting to disable the Roxen extended environment variables.

  • Javascript support: Don't output empty <script> tags. Marginally smarter quoting.

Minor improvements:

  • Administration interface: Updated the default logging format to the Combined Log Format which extends Common Log Format with two fields. The proposed format with extended usage info is changed accordingly. It also uses $ip-number instead of $host by default, to avoid the DNS overhead.

  • Administration interface: The Add modules page now shows the site name, to avoid adding modules to the wrong site by mistake.

  • Administration interface: A bit nicer sort order for threads in Tasks/Debug Information/Thread backtrace, e.g. the backend thread is always listed first.

  • Administration interface: Improved display of Unix sockets in Tasks/Debug Information/Open files.

  • Administration interface: The binary distribution identifier is now shown at the bottom of the pages.

  • Core: Optimizations in the protocol cache, and in the vary callback system in particular.

  • Core: The image cache now allows protocol caching of authenticated images, which is made possible by the new arg cache implementation.

  • Core: Speed up accepting of connections.

  • Core: Improved the p-code codec to handle references to arbitrary Pike modules.

  • Core: The sample start script in the tools directory now has a "status" command to query whether the Roxen instance is running or not. It can also be configured to handle several instances by keeping their configuration directories in a special "configuration collection" directory.

  • Core: Added an experimental mode where the RAM cache retention policy is based on the time to create the entries. This is enabled by the define TIME_BASED_CACHE.

  • Core: Log a warning if a background job takes more than one minute.

  • Core: Roxen modules are now always loaded in the same order.

  • Core: There is now a bat file bin\mysql_client_nt.bat to make it simpler to start a MySQL command line client against Roxen's MySQL process on Windows.

  • RXML tags: The <nocache> tag now properly disables protocol and client caching, just like the <cache> tag does by default. It has also gotten the attributes enable-client-cache and enable-protocol-cache that works like in the <cache> tag, to control this behavior.

  • RXML tags: <remove-cookie> no longer requires the cookie to exist.