Foswiki Upgrade Guide

This guide covers upgrading from a previous version of Foswiki to Foswiki 2.x.

ALERT! This upgrade procedure is used to upgrade a to a new major version of Foswiki (1.x to 2.x) or from TWiki. Generally, upgrades of a minor Foswiki version (2.0 to 2.1), or Foswiki patch releases (2.1.3 to 2.1.4) can be done by using the -upgrade- version of the Foswiki package, and follow the Release Notes for the new release.

On this page:

Overview

Foswiki is a fork from TWiki 4.2.3. (TWiki® is a trademark of Peter Thoeny.) Note that newer versions of Foswiki have not directly tracked newer versions of TWiki, and some divergence has occurred. Foswiki 2.0 diverges significantly in some areas.

If you are upgrading from TWiki to Foswiki, please refer to Foswiki:Support.UpgradingFromOlderTWikiReleases.

Upgrade requirements

Upgrading to a new patch or minor release

To upgrade to a new patch release — for example, from Foswiki 1.1.0 to 1.1.2 — an upgrade package can be used that will not overwrite typical customizations. Unless otherwise stated in the release notes, we do not recommend upgrading between major or minor versions using the patch (For ex. 1.1.9 to 2.0). A re-installation is recommended.

For patch releases you will find a brief upgrade procedure on the download page for the release. Follow this procedure to upgrade to the patch release. It may contain important steps that are unique to each patch release (for example, some configure settings may need to be changed).

The following files are excluded from the upgrade packages. Any other files will be overwritten! If you have tailored any other files/topics shipped with Foswiki, you will need to either restore them after applying the upgrade, or tailor the upgrade to avoid overwritting the files.

Main web files not shipped in upgrade package

  • data/Main/AdminGroup.txt
  • data/Main/AdminUser.txt
  • data/Main/NobodyGroup.txt
  • data/Main/PatternSkinUserViewTemplate.txt
  • data/Main/ProjectContributor.txt
  • data/Main/RegistrationAgent.txt
  • data/Main/SitePreferences.txt
  • data/Main/UnknownUser.txt
  • data/Main/UserHomepageHeader.txt
  • data/Main/UserListByDateJoined.txt
  • data/Main/UserListByLocation.txt
  • data/Main/UserListHeader.txt
  • data/Main/UserList.txt
  • data/Main/WebAtom.txt
  • data/Main/WebChanges.txt
  • data/Main/WebCreateNewTopic.txt
  • data/Main/WebHome.txt
  • data/Main/WebIndex.txt
  • data/Main/WebLeftBarExample.txt
  • data/Main/WebNotify.txt
  • data/Main/WebPreferences.txt
  • data/Main/WebRss.txt
  • data/Main/WebSearchAdvanced.txt
  • data/Main/WebSearch.txt
  • data/Main/WebTopicList.txt
  • data/Main/WikiGuest.txt

Sandbox web files not shipped in upgrade package

  • data/Sandbox/CommentPluginExampleComments.txt
  • data/Sandbox/CommentPluginExamples.txt
  • data/Sandbox/CommentPluginTemplateExample.txt
  • data/Sandbox/CompareRevisionsAddOnDemoTopic.txt
  • data/Sandbox/PluginTestEmptyPlugin.txt
  • data/Sandbox/WebAtom.txt
  • data/Sandbox/WebChanges.txt
  • data/Sandbox/WebCreateNewTopic.txt
  • data/Sandbox/WebHome.txt
  • data/Sandbox/WebIndex.txt
  • data/Sandbox/WebLeftBarExample.txt
  • data/Sandbox/WebNotify.txt
  • data/Sandbox/WebPreferences.txt
  • data/Sandbox/WebRss.txt
  • data/Sandbox/WebSearchAdvanced.txt
  • data/Sandbox/WebSearch.txt
  • data/Sandbox/WebTopicList.txt

System web files not shipped in upgrade package

  • data/System/InterWikis.txt
  • data/System/NewUserTemplate.txt
  • data/System/WebPreferences.txt
  • data/System/WebTopicEditTemplate.txt

Other miscellaneous web files not shipped in upgrade package

  • data/_default/WebHome.txt
  • data/_default/WebLeftBar.txt
  • data/_default/WebPreferences.txt
  • data/Trash/TrashAttachment.txt
  • data/Trash/WebPreferences.txt
  • data/TWiki/WebPreferences.txt

Other files not shipped in upgrade package

  • index.html
  • robots.txt

TIP If you use the Foswiki PageCaching feature, be sure to refresh the cache after upgrading to a new Foswiki version. Visit your site with the Query parameter ?refresh=all

Upgrade procedure: upgrading to a new major version

The following is a high level view of the upgrade procedure:

  1. Before the upgrade, plan for client cache management.
  2. Prepare for all upgrade steps.
  3. Install the new Foswiki version and configure it with the same settings as the old version.
  4. Install any additional extensions (Plugins) used by your old installation. Make sure to use the latest Foswiki versions.
  5. Convert all the non-default webs from the old installation to the new one. (Encoding and Store changes)
  6. Convert the users, groups, and site customizations from the old installation to the Main web in the new installation, including all user topics.
  7. Apply preferences from the old installation.
  8. Apply your site customizations: skin, logos, menu bars, forms for personal information, and so forth.
  9. Validate your Wiki applications and other key functionality.
  10. Switch your production site from the old installation to the new installation.

More details for each step appear in the following sections. The steps may need to be modified or otherwise tailored with specifics for your installation. In particular, you must take care to preserve any special configuration or customizations you have made, especially if you have modified any of the default software files or system topics that are contained within the installation package.

For purposes of discussion, the following conventions are used:

After the upgrade, in the new installation, the Main web is used for user topics and site preferences, and the System web is used to hold documentation and default preferences.

The configure page mentioned in this document is accessible via your web browser at http://yourdomain/<newwiki>/bin/configure .

Before the upgrade

Managing caches of static .js, .css files:

If you are using Expires tags, (you should be!) it is very important to take the longest expiration time into consideration. Clients will locally cache JavaScript and CSS until the time expires, unless they clear their cache. There are significant changes to the JavaScript and CSS files shipped with Foswiki 2.0. Clients using locally cached data will not operate correctly.

If these steps are not done, users will have to clear their browser cache to get the most recent CSS and JavaScript..

Prepare for all upgrade steps

Download the Foswiki distribution from the following location: https://foswiki.org/Download — if you are installing your extensions manually, also download them from the repository where they are stored. (Default extensions are included in the Foswiki distribution).

Review Foswiki:System.SystemRequirements and Install missing Perl modules using your local package manager or CPAN. Foswiki 2.0 no longer ships with CPAN modules. If you have access to the command line on the server, you can test for missing dependencies by running: perl tools/dependencies

Review the Release Notes and learn about the differences between your old installation and the new release to which you are upgrading. Take note of any areas that affect your site and what special steps you may need to take.

Check that all the extensions (plugins, contribs, skins) used by your old installation are available with the new release. Familiarize yourself with any new behaviour that you will have to adapt to or any configuration changes you will have to perform.

Choose the character encoding to be used in your site. Note that the 1.x versions of Foswiki make no attempt to enforce the character encoding. Topics contain whatever the user may have pasted into them. If you have windows users, it is highly likely that your old installation is using cp-1252 instead of iso-8859-1. cp-1252 contains windows specific high-ASCII characters including typographic "smart" quotes, alternative bullets and the emdash character. For more information see Foswiki:Support.Utf8MigrationConsiderations.

WARNING If you do not use the utf-8 {Store}{Encoding} and you intend to use, (or have existing) high-bit characters in attachment filenames (such as umlauts and accents) then links to these attachments on Foswiki pages will not work. This is because Foswiki works internally using UNICODE, but the store saves files to disk using your chosen {Store}{Encoding}. The solution is to convert your store to UTF8 at the earliest opportunity.

You must match the prior encoding, or convert old data to the new encoding if you intend to use topics created on an older version of Foswiki. There are two common use cases:

Choose your desired Store. Foswiki ships with two native stores.

If you are using authentication, prepare a test plan to verify that your authentication mechanism is working correctly. Make sure you are able to test logins by a sufficient sample of users to cover all categories of users of your site. For example, users of various groups may need to be tested. In particular, ensure you test that non-admin users cannot access topics restricted to admins.

Identify all essential Wiki topics and Wiki applications that must be fully functional upon completion of the upgrade. Prepare a test plan to verify their functionality. If you are using access controls, ensure that the test plan will adequately test all categories and groups of users of your site.

If your testing will require a test environment to be set up, ensure that it is ready, with any required support infrastructure (for example, testbed authentication servers). If you need to be able to login with different users in different categories and groups, ensure that you have the required login information ready, or you have testers from those groups available to perform the required test cases.

Installation

Follow the installation instructions in INSTALL.html, located in the root of the new installation, or online at Foswiki:System.InstallationGuide. Install the new release in a new directory. Do not install on top of the old release.

Test your newly-installed Foswiki site and ensure that its basic functionality works: viewing and editing topics (you can try creating and editing a topic in the Sandbox web).

ALERT! Caution: If you intend to copy data from an older installation without using bulk_copy.pl to change Stores, you should select the RcsStoreContrib in the configuration. Once topic history has been created with the wrong store, it has to either be removed, or old data should be migrated with bulk_copy.pl. If Foswiki encounters mixed RCS and PlainFile topic history, it will "die" to prevent topic history corruption.

If you want to run Foswiki 2.0 in parallel with Foswiki 1.1.9, you can do this safely if the following conditions are met:
To make it easier to follow the subsequent steps, you can view this upgrade guide using your new Foswiki site by entering System.UpgradeGuide into the "Jump" text box on the top right of any topic. By doing this instead of using the UpgradeGuide.html file from the distribution, you will be able to use the embedded hyperlinks to jump directly to the referenced pages.

Install extensions

Install all of the extensions that were installed in your old site. In particular, start with any extensions required for the authentication and authorization methods you use (if any). You can use the Install, Update or Remove extensions tab in the Extensions section of the configure page to review installed extensions, search for extensions or all available extensions and configure extensions from the Foswiki:Extensions repository. You can also install extensions manually; see the instructions on the extension's web page from where you obtained the extension (for Foswiki extensions, on foswiki.org).

Check the plugin topics from your old Foswiki installation and transfer the plugin settings to the Main.SitePreferences topic in your new Foswiki site, prefixing each setting with the name of the plugin in uppercase followed by an underscore. For example, to copy over the DEFAULT_TYPE setting from the CommentPlugin topic in the old site to the new site, copy the value to a COMMENTPLUGIN_DEFAULT_TYPE setting in the Main.SitePreferences topic in the new site.

Commonly-customized plugin settings include the following:

Activate, and if required, configure the installed extensions in configure.

Copy parts of the working directory

The working directory contains some critical information for some extensions, found below the foswiki/working/work_areas directory. Extensions use it to store persistent information critical to operation. For example, the MailerContrib directory contains the timestamps of the last notification email run per web. If not copied, the next mailnotify run will notify all recorded changes. This is the most common data that should be copied. Review other non-default extensions to determine if anything else should be copied.

Migrate .htpasswd file

If used, the .htpasswd file may require conversion. If users have registered using "high ascii" characters in their WikiNames (e.g. Ä, ä, Ë, ë, Ï, ï, Ö, ö, Ü, ü) then conversion will be required. Foswiki does not provide a tool to convert .htpasswd. Consider command line tools such as iconv or recode. See this StackOverflow article for more details.

If you are using an external authentication source, such as LDAP, this step does not apply.

Alternative 1: Convert the data using tools/bulk_copy.pl

This is the default way to migrate your system. Note cautions below about hidden files and performance. Assume that you have the following setup: Use the bulk_copy.pl tool to migrate your data: 

cd /var/www/f120/tools
perl bulk_copy.pl --xweb System --xweb _default --xweb _empty --latest '*.WebStatistics' /var/www/f119/bin /var/www/f120/bin

This will copy all webs, topics and attachments except for the contents of the System web. This is the recommended solution.

Note that bulk_copy.pl has limitations! It uses the Foswiki Store API to copy topic and attachments. If the store doesn't recognize a topic or attachment, it won't be copied.

ALERT! bulk_copy.pl executes the code and APIs of the older Foswiki release. Older versions of Foswiki may fail on the latest versions of Perl due to language changes. If you encounter this issue, you could try an upgrade of the 1.1.x system to Foswiki 1.1.10, or use the CharsetConverterContrib to complete the migration.

ALERT! bulk_copy.pl is not compatible with Foswiki 1.0.x versions. If you are copying data from Foswiki 1.0.x, there are two options:
  1. Use CharsetConverterContrib to convert the data "in-place" on Foswiki 2.x -or-
  2. Convert to Foswiki 1.1.10 first, and then convert to Foswiki 2.x.
The first option is faster and less complex.

Before proceeding with conversion, verify that the above limitations are not applicable to your installation. If they are, you will currently need to:

Alternative 2: Convert data using CharsetConverterContrib

These steps can be used to manually migrate data when not changing the Store type. This is currently the only way to deal with hidden files.

Copy content from non-default webs in old installation to the new installation

Be sure to select an "RCS Store" RcsWrap or RcsLite on the new installation. The PlainFile store is not compatible with topic history written on previous versions of Foswiki. If you have created or updated topics using PlainFileStore, you should either start over, or plan to to remove all ,pfv directories from the system so that there is no history in the PlainFileStore format.

Copy your local webs over to the data and pub directories of the new installation. Do not copy the default webs: <old_system_web> System, Main, Trash, Sandbox, _default, and _empty.
  • It may be preferable to copy and convert one web at a time.
  • Make sure the data and pub directories, as well as the files within them, are readable and writeable by the web server user.
  • Note: Foswiki's WebChanges topics depend on the file timestamp. If you touch the .txt files make sure to preserve the timestamp, or change them in the same chronological order as the old file timestamps.

If needed, convert the character encoding of each web before attempting to access it from the web. The following CharsetConverterContrib command will "inspect" a single web and report any conversion issues. Remove the -i option to proceed with the actual conversion. ALERT! Do not run the actual conversion more than once on a web!
  • perl convert_charset.pl -i -web=<Webname> -encoding=cp-1252

Verify that existing topics are operational and (if you converted to UTF-8) that any international characters have been properly converted and are displayed correctly.

Copy users, user topics, and site customizations to Main web

Copy all topics and attachments from <old_users_web>: copy all files from <oldwiki>/data/<old_users_web>/ to <newwiki>/data/Main/, and copy all files from <oldwiki>/pub/<old_users_web>/ to <newwiki>/pub/Main/ . Do not overwrite any topics already present in the <newwiki>/data/Main/ directory.
  • In addition to all the user topics, if you have created <old_users_web>.NewUserTemplate in the old installation, this step will copy over your template for user topics to the new installation.
  • Ensure that the topic defining the admin group in your old installation is copied over. The admin group is defined in the Security setup pane of the configure page, in the {SuperAdminGroup} setting (visible when Expert mode is enabled). You can do either of the following:
    • Set the {SuperAdminGroup} setting in your new installation to the old admin group.
    • Move the contents of the old admin group to the new admin group. To avoid having to change all references to the old admin group, you must still keep the old admin group defined: set it so its only member is the new admin group, and the new admin group is the only user who can change or rename the old admin group topic.
  • If your old installation did not customize {LocalSitePreferences} on the configure page, or if you did customize {LocalSitePreferences} but kept your site preferences within the <old_users_web> web, then this step will also copy over your site preferences to the new installation.

Use the CharsetConverterContrib to convert the new Main as described above.

Copy over any topics and attachments you want to preserve from the Sandbox web in the old installation: copy the desired files from <oldwiki>/data/Sandbox/ to <newwiki>/data/Sandbox and from <oldwiki>/pub/Sandbox/ to <newwiki>/pub/Sandbox . Some pages you may wish to preserve are the WebHome topic and the WebLeftBar topic (if you had created it in the old wiki installation). The Sandbox web often contains work-in-progress topics that users will want to keep.

Make sure the data and pub directories, as well as the files within them, are readable and writeable by the web server user.

Main changes for older Foswiki installations

For upgrades from an older Foswiki installation:

Convert empty DENY ACLs to ALLOW * wildcards

By default, empty DENYTOPIC rules will be ignored by Foswiki 2.0. You must change them to the equivalent ALLOWTOPIC * rules. The tools/convertTopicSettings.pl utility will scan the Webs & Topics, and will perform several optional conversions on the topics.
Get help text for convertTopicSettings
perl tools/convertTopicSettings.pl -help
Scan all webs / topics, report any topics with empty DENY rules
perl tools/convertTopicSettings.pl
Replace all empty DENY rules with ALLOW * wildcards
perl tools/convertTopicSettings.pl -fixdeny -update

The following two options are also available, but are not yet recommended.
Same, but convert all ACLs into META settings from inline topic settings, for just the Sandbox web
perl tools/convertTopicSettings.pl -fixdeny -convert -update Sandbox
Convert ALL settings into META settings, not just ACLs, for the Sandbox and Customer webs
perl tools/convertTopicSettings.pl -fixdeny -convert -all -update Sandbox Customer

When convertTopicSettings saves the modified topics, they will be saved by user UnknownUser.

DataForms Applications

If your site uses DataForms that used non-Ascii field names, the form data will require manual migration, or you must enable {LegacyFormfieldNames} in the configuration.

If you do not enable {LegacyFormfieldNames}, then you will need to find and update the META:FIELD definitions in the topics. This would need to be done external to Foswiki. 
  %META:FIELD{name="Fhler" title="Fühler" value="123"}%
would need to be changed to 
  %META:FIELD{name="Fühler" title="Fühler" value="123"}%

Apply preferences from old installation

If you have not already set your desired site-wide preferences, as described in the section " Set Foswiki Preferences" in the InstallationGuide, then set your preferences. The location of your site preferences is specified in the Miscellaneous settings pane of the configure page, in the {LocalSitePreferences} setting (visible when Expert mode is enabled) — the default location is Main.SitePreferences. Copy any customized preferences from the site preferences topic in your old installation to the site preferences topic in the new installation. (Note you may have already copied over your customized preferences when you transfered the contents of the <old_users_web> web.) (These should have been copied when your Main was migrated.)

If, in your old installation, you customized the default preferences in <old_system_web>.DefaultPreferences, then transfer your customizations from this topic to the SitePreferences topic instead (i.e. the topic specified in your {LocalSitePreferences} setting), so that your customizations will not get overwritten on the next upgrade.

If you have any old extensions that use settings from the "System.<Extension>" topic, these settings should also be copied to the SitePreferences. Topics in the System should never be edited!

Apply additional site customizations

Modify skin with customizations for your site

If you did not already customize the appearance of your new installation, as described in the section " Customize the appearance of your Foswiki site" in the InstallationGuide, then reapply the customizations from your old installation to the new one. Ensure you transfer over any skin templates — .tmpl files, or topics referred to using VIEW_TEMPLATE or EDIT_TEMPLATE preferences — you need. Also ensure you transfer any style sheets or Javascript files required.

Customize pages for managing personal information

In your new installation, default copies of the following topics were installed:

If you customized these topics in your old installation, transfer the changes to these topics in the new installation. Use the corresponding files in the <oldwiki>/<old_system_web>/ directory as a reference.

Check your configuration and installation

Configure provides some tools to validate your installation. They should be run as the web server userid: 
cd <path-to-foswiki-installation>
sudo -u www-data tools/configure -check
sudo -u www-data perl tools/configure -check {DataDir} -method validate_permissions
sudo -u www-data perl tools/configure -check {PubDir} -method validate_permissions
 ...  also can be run on: 
    {LocalesDir} {ScriptDir}  {TemplateDir} {ToolsDir} {WorkingDir}

Validate your Wiki applications and other key functionality

Execute your test plan to validate the Wiki applications and other key functionality that need to be up and running after the upgrade.

Execute your test plans for authentication and authorization. Test that users that you have transferred from the old installation can login with any problems, and that access controls work appropriately: check that users are able to view and edit pages for which they have access, and are denied permission to view or edit pages for which they do not have access. Also check that pages restricted to the admin group are not accessible by non-admin users, and that administrators continue to have access.

Switch your production site from the old installation to the new installation

ALERT! If you are converting from RCS to PlainFile store, you must not repeat any copy step from the old to the new version once you've run the conversion.

If you had been running your old installation in parallel with the new one during a test phase, then disable your old installation, and repeat the steps:

Change your web server configuration so that the new installation is accessible to all of your users, and so the old installation is no longer accessible.

Change your web server configuration so that the new installation is accessible using the same URL prefix as your old installation. For purposes of discussion, assume that your old installation is accessible from http://yourdomain/wiki/. You can use one of the following approaches to make the new installation accessible using the same URL prefix:

Re-execute your test plan to verify that your newly-upgraded site is accessible to your users, and that all authentication and authorization mechanisms work as expected (including denying access to those who are not authorized).

Re-execute your test plan to verify that your Wiki applications and other key functionality work as intended.

Related Topics: AdminDocumentationCategory, Foswiki:Support.InstallationGuide, Foswiki:Support.InstallingOnSpecificPlatforms, Foswiki:Support.ApacheConfigGenerator, Foswiki:Support.SettingFileAccessRightsLinuxUnix, ReleaseNotes01x00, ReleaseNotes01x01