Uploaded image for project: 'Hue'
  1. Hue
  2. HUE-2446

Migrating from CDH 4.7 to CDH 5.0.1+/Hue 3.5+ will fail

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.5.0, 3.5.0, 3.6.0, 3.7.0
    • Fix Version/s: None
    • Component/s: core.backend
    • Labels:
      None

      Description

      In CDH 4.7.0 in the commit c61ff8, we backported apps/oozie/src/oozie/migrations/0025_change_examples_path_format.py and renamed it to 0022_change_examples_path_format.py. When a user then upgrades their Hue instance to CDH 5.0.1+/Hue 3.5.0+, the upgrade process removes the 0022_change_examples_path_format.py file. This will result in the user receiving a ghost migration error like this:

      Traceback (most recent call last):
        File "./build/env/bin/hue", line 9, in <module>
          load_entry_point('desktop==3.6.0', 'console_scripts', 'hue')()
        File "/opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/hue/desktop/core/src/desktop/manage_entry.py", line 60, in entry
          execute_manager(settings)
        File "/opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/hue/build/env/lib/python2.6/site-packages/Django-1.4.5-py2.6.egg/django/core/management/__init__.py", line 459, in execute_manager
          utility.execute()
        File "/opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/hue/build/env/lib/python2.6/site-packages/Django-1.4.5-py2.6.egg/django/core/management/__init__.py", line 382, in execute
          self.fetch_command(subcommand).run_from_argv(self.argv)
        File "/opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/hue/build/env/lib/python2.6/site-packages/Django-1.4.5-py2.6.egg/django/core/management/base.py", line 196, in run_from_argv
          self.execute(*args, **options.__dict__)
        File "/opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/hue/build/env/lib/python2.6/site-packages/Django-1.4.5-py2.6.egg/django/core/management/base.py", line 232, in execute
          output = self.handle(*args, **options)
        File "/opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/hue/build/env/lib/python2.6/site-packages/South-0.8.2-py2.6.egg/south/management/commands/migrate.py", line 111, in handle
          ignore_ghosts = ignore_ghosts,
        File "/opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/hue/build/env/lib/python2.6/site-packages/South-0.8.2-py2.6.egg/south/migration/__init__.py", line 200, in migrate_app
          applied_all = check_migration_histories(applied_all, delete_ghosts, ignore_ghosts)
        File "/opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/hue/build/env/lib/python2.6/site-packages/South-0.8.2-py2.6.egg/south/migration/__init__.py", line 95, in check_migration_histories
          raise exceptions.GhostMigrations(ghosts)
      south.exceptions.GhostMigrations: 
      
       ! These migrations are in the database but not on disk:
          <oozie: 0022_change_examples_path_format>
       ! I'm not trusting myself; either fix this yourself by fiddling
       ! with the south_migrationhistory table, or pass --delete-ghost-migrations
       ! to South to have it delete ALL of these records (this may not be good).
      

      This particular migration appears to be idempotent, so this gives us a couple options to fix this particular bug:

      • We could just re-add the 0022_change_examples_path_format.py file to the Hue master branch, and all of the older branches. We added running our migrations with -merge in 11582cbd, so this should be safe. However, always running with -merge could someday cause problems for a non-idempotent migration.
      • We could document that before you upgrade from 4.7.0 you should run hue migrate oozie 0021 to revert that migration.
      • We could document that this particular migration should be deleted. This can be done with this script:
      % ./build/env/bin/hue shell
      >>> from south.models import MigrationHistory
      >>> try:
      ...     m = MigrationHistory.objects.filter(app_name='oozie', migration='0022_change_examples_path_format').get()
      ... except MigrationHistory.DoesNotExist:
      ...     pass
      ... else:
      ...     m.delete()
      

        Attachments

          Activity

            People

            • Assignee:
              erickt Erick Tryzelaar
              Reporter:
              erickt Erick Tryzelaar
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: