update failed: 503 service unavailable

Does your proxy-server.conf file have the line "allow_account_management = true" in the "[app:proxy-server]" section? Are there any entries from the proxy-server above the lines you posted indicating the 403 given to the auth-server?

I have double checked the following conf files:

1 /etc/swift/proxy-server.conf

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true <==no problem

2 /etc/swift/auth-server and /etc/swift/proxy-server.conf

/etc/swift/proxy-server
[DEFAULT]
bind_port = 8080 -----------------------------------------
user = bella |
                                                                             |
/etc/swift/auth-server.conf |=>has the same bind_port
[app:auth-server] |
use = egg:swift#auth |
default_cluster_url = http://127.0.0.1:8080/v1 ------

3 ll -d /etc/swift
drwxr-xr-x 6 bella bella 4096 2011-03-10 11:13 /etc/swift/ <=bella is the user id

4 /etc/swift/account-server/[1-4].conf , /etc/swift/container-server/[1-4].conf, /etc/swift/object-server/[1-4].conf

mount_check = false

5 full content of /etc/swift/proxy-server.conf is:

DEFAULT]
bind_port = 8080
user = bella

[pipeline:main]
# For DevAuth:
pipeline = healthcheck cache auth proxy-server
# For Swauth:
# pipeline = healthcheck cache swauth proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true

# Only needed for DevAuth
[filter:auth]
use = egg:swift#auth

# Only needed for Swauth
#[filter:swauth]
#use = egg:swift#swauth
# Highly recommended to change this.
super_admin_key = filestore

[filter:healthcheck]
use = egg:swift#healthcheck

6 full content of /etc/swift/auth-server.conf
[DEFAULT]
user = bella

[pipeline:main]
pipeline = auth-server

[app:auth-server]
use = egg:swift#auth
default_cluster_url = http://127.0.0.1:8080/v1
# Highly recommended to change this.
super_admin_key = filestore

[filter:cache]
use = egg:swift#memcache

7 other entries from /var/log/syslog about the proxy-server:

Mar 9 19:31:23 ubuntuswift swift CRITICAL: THERE ARE ACCOUNTS IN YOUR auth.db THAT DO NOT BEGIN WITH YOUR NEW RESELLER PREFIX OF "AUTH". YOU HAVE A FEW OPTIONS: 1) RUN "swift-auth-update-reseller-prefixes /tmp/swift_test_auth_2bba374d0b7d4699875e443a24abbbf5/auth.db AUTH", "swift-init auth-server restart", AND "swift-auth-recreate-accounts -K ..." TO CREATE FRESH ACCOUNTS. OR 2) REMOVE /tmp/swift_test_auth_2bba374d0b7d4699875e443a24abbbf5/auth.db, RUN "swift-init auth-server restart", AND RUN "swift-auth-add-user ..." TO CREATE BRAND NEW ACCOUNTS THAT WAY. OR 3) ADD "reseller_prefix = " (WITHOUT THE QUOTES) TO YOUR proxy-server.conf IN THE [filter:auth] SECTION AND TO YOUR auth-server.conf IN THE [app:auth-server] SECTION AND RUN "swift-init proxy-server restart" AND "swift-init auth-server restart" TO REVERT BACK TO YOUR PREVIOUS RESELLER PREFIX. SINCE YOUR PREVIOUS RESELLER PREFIX WAS AN EMPTY STRING, IT IS NOT RECOMMENDED TO PERFORM OPTION 3 AS THAT WOULD MAKE SUPPORTING MULTIPLE RESELLERS MORE DIFFICULT.

thanks a lot, gholt :)

Ah, you have a really old auth.db from early dev. I get it. Let me know if you have trouble with upgrade instructions posted in the log [reposted here for future searchers]:

Mar 9 19:31:23 ubuntuswift swift CRITICAL:

THERE ARE ACCOUNTS IN YOUR auth.db THAT DO NOT BEGIN WITH YOUR NEW RESELLER PREFIX OF "AUTH". YOU HAVE A FEW OPTIONS:

1) RUN "swift-auth-update-reseller-prefixes /tmp/swift_test_auth_2bba374d0b7d4699875e443a24abbbf5/auth.db AUTH", "swift-init auth-server restart", AND "swift-auth-recreate-accounts -K ..." TO CREATE FRESH ACCOUNTS. OR

2) REMOVE /tmp/swift_test_auth_2bba374d0b7d4699875e443a24abbbf5/auth.db, RUN "swift-init auth-server restart", AND RUN "swift-auth-add-user ..." TO CREATE BRAND NEW ACCOUNTS THAT WAY. OR

3) ADD "reseller_prefix = " (WITHOUT THE QUOTES) TO YOUR proxy-server.conf IN THE [filter:auth] SECTION AND TO YOUR auth-server.conf IN THE [app:auth-server] SECTION AND RUN "swift-init proxy-server restart" AND "swift-init auth-server restart" TO REVERT BACK TO YOUR PREVIOUS RESELLER PREFIX.

SINCE YOUR PREVIOUS RESELLER PREFIX WAS AN EMPTY STRING, IT IS NOT RECOMMENDED TO PERFORM OPTION 3 AS THAT WOULD MAKE SUPPORTING MULTIPLE RESELLERS MORE DIFFICULT.

Good afternoon, gholt. I have tried the first method you mentioned in the post, but when I typed in command:

swift-auth-update-reseller-prefixes /tmp/swift_test_auth_2bba374d0b7d4699875e443a24abbbf5/auth.db AUTH

it returned the following message:

Updating /tmp/swift_test_auth_2bba374d0b7d4699875e443a24abbbf5/auth.db
Traceback (most recent call last):
  File "/usr/local/bin/swift-auth-update-reseller-prefixes", line 7, in <module>
    execfile(__file__)
  File "/home/bella/swift/trunk/bin/swift-auth-update-reseller-prefixes", line 32, in <module>
    conn = get_db_connection(db)
  File "/home/bella/swift/trunk/swift/common/db.py", line 147, in get_db_connection
    timeout=timeout)
swift.common.db.DatabaseConnectionError: DB connection error (/tmp/swift_test_auth_2bba374d0b7d4699875e443a24abbbf5/auth.db, 30):
Traceback (most recent call last):
  File "/home/bella/swift/trunk/swift/common/db.py", line 129, in get_db_connection
    factory=GreenDBConnection, timeout=timeout)
  File "/home/bella/swift/trunk/swift/common/db.py", line 68, in __init__
    sqlite3.Connection.__init__(self, *args, **kwargs)
OperationalError: unable to open database file

then I tried the second method , and found that the file /tmp/swift_test_auth_2bba374d0b7d4699875e443a24abbbf5/auth.db
dosen't exist.

what should I do next? By the way,what is reseller prefix? did I mis-config something?

thank you very much, gholt! :)

hello again, gholt.

I tried the command :
swift-init auth-server restart

and it retured:
Stopping auth-server pid: 2597 signal: 15
Unable to increase file descriptor limit. Running as non-root?
Starting auth-server

and then I tried the command :
swift-auth-add-user -K filestore -a test tester testing

then it return the same 503 error message:
Update failed: 503 Service Unavailable

I checked the /var/log/syslog, it says:

Mar 10 14:07:21 ubuntuswift auth-server SIGTERM received
Mar 10 14:07:21 ubuntuswift auth-server Exited
Mar 10 14:07:21 ubuntuswift auth-server Started child 2634
Mar 10 14:09:01 ubuntuswift CRON[2640]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 | xargs -n 200 -r -0 rm)
Mar 10 14:10:37 ubuntuswift auth-server ERROR attempting to create account http://127.0.0.1:8080/v1/AUTH_c6d0a529d3684829bb903d027fde9e4b: 403 Forbidden
Mar 10 14:10:37 ubuntuswift auth-server FAILED create_user('test', 'tester', _, True, False) [0.04]
Mar 10 14:10:37 ubuntuswift auth-server 127.0.0.1 - - [10/Mar/2011:06:10:37 +0000] "PUT /account/test/tester HTTP/1.0" 503 - "-" "-" - - - - - - - - - "-" "127.0.0.1" "-" 0.0466

also there are entries about proxy-server in /var/log/syslog:

Mar 10 13:43:43 ubuntuswift swift CRITICAL: THERE ARE ACCOUNTS IN YOUR auth.db THAT DO NOT BEGIN WITH YOUR NEW RESELLER PREFIX OF "AUTH". YOU HAVE A FEW OPTIONS: 1) RUN "swift-auth-update-reseller-prefixes /tmp/swift_test_auth_b2bbd80c0da24a6aaec5edcf05fd1a66/auth.db AUTH", "swift-init auth-server restart", AND "swift-auth-recreate-accounts -K ..." TO CREATE FRESH ACCOUNTS. OR 2) REMOVE /tmp/swift_test_auth_b2bbd80c0da24a6aaec5edcf05fd1a66/auth.db, RUN "swift-init auth-server restart", AND RUN "swift-auth-add-user ..." TO CREATE BRAND NEW ACCOUNTS THAT WAY. OR 3) ADD "reseller_prefix = " (WITHOUT THE QUOTES) TO YOUR proxy-server.conf IN THE [filter:auth] SECTION AND TO YOUR auth-server.conf IN THE [app:auth-server] SECTION AND RUN "swift-init proxy-server restart" AND "swift-init auth-server restart" TO REVERT BACK TO YOUR PREVIOUS RESELLER PREFIX. SINCE YOUR PREVIOUS RESELLER PREFIX WAS AN EMPTY STRING, IT IS NOT RECOMMENDED TO PERFORM OPTION 3 AS THAT WOULD MAKE SUPPORTING MULTIPLE RESELLERS MORE DIFFICULT.

and the file /tmp/swift_test_auth_b2bbd80c0da24a6aaec5edcf05fd1a66/auth.db doesn't exist, but I found that there are many empty folders under /tmp like /tmp/tmpzXPaFQ/

I use the command locate auth.db, there is only one file: /etc/swift/auth.db

I have tried Swauth , but it still report the error message "ubuntuswift swift CRITICAL: THERE ARE ACCOUNTS IN YOUR auth.db THAT DO NOT BEGIN WITH YOUR NEW RESELLER PREFIX OF "AUTH"...", I really don't know how to resolve this problem...

If you are now using Swauth, it does not need the auth-server process, that's only required for DevAuth. The /tmp/swift_test_auth_b2bbd80c0da24a6aaec5edcf05fd1a66/ path confuses me as well, do you have a swift_dir = /tmp/swift_test_auth_b2bbd80c0da24a6aaec5edcf05fd1a66/ somewhere in your configs?

If you can, can you log into IRC, maybe we can work it out faster there? chat.freenode.net and #openstack

I assume you're just working on a test/dev system. Since it has been a while between updates to your system, it might be best to refresh and work through the setup again.

Thank you a lot, gholt.

I tried the folling Swauth authentication following the instruction and it still reported the same "update failed:503 message"
error.

I have checked my conf fiiles again, the /tmp/swift_test_auth_b2bbd80c0da24a6aaec5edcf05fd1a66/ path doesn't exsit.

I don't have IRC account, can we discuss on msn, my msn account is <email address hidden>

And yes, after tried all methods but failed, I am trying to set up swift again , and not yet finished .

Thank you a lot, gholt.

I tried the folling Swauth authentication following the instruction and it still reported the same "update failed:503 message"
error.

I have checked my conf fiiles again, the /tmp/swift_test_auth_b2bbd80c0da24a6aaec5edcf05fd1a66/ path doesn't exsit.

I don't have IRC account, can we discuss on msn, my msn account is <email address hidden>

And yes, after tried all methods but failed, I am trying to set up swift again , and not yet finished .

Good afternoon, gholt.

I re-deployed swift and no more errors "update failed:503 service unavailable", I guess maybe the new version swift code resolved this problem itself.

By the way, I checked the /var/log/syslog, it still has entries of error about proxy-server, like "THERE ARE ACCOUNTS IN YOUR auth.db THAT DO NOT BEGIN WITH YOUR NEW RESELLER PREFIX OF "AUTH". ". I think this error message is irrelavant with "update failed:503 service unavailable", and I noticed that this message appeard after I typed in the command "cd ~/swift/trunk; ./.unittests".

Thanks a lot, gholt.

Dear gholt,

     I am interested in consistent hashing and the ring mechanism in openstack swift, when I google some articles on the internet, I found a serious articles you have written about consistent hashing on your blog.

     But I have a question about the third code section in second article(http://tlohg.wordpress.com/2011/02/08/building-a-consistent-hashing-ring-part-2/), when we add a new node, the vnode2node mapping is almost completely changed, why you only adjust the vnode to new node mapping, and I don't think that's the right mapping, do you?

     I copy your code here:

from bisect import bisect_left
from hashlib import md5
from struct import unpack_from

NODE_COUNT = 100
DATA_ID_COUNT = 10000000
VNODE_COUNT = 1000

vnode_range_starts = []
vnode2node = []
for vnode_id in xrange(VNODE_COUNT):
    vnode_range_starts.append(DATA_ID_COUNT /
                              VNODE_COUNT * vnode_id)
    vnode2node.append(vnode_id % NODE_COUNT)
new_vnode2node = list(vnode2node)
new_node_id = NODE_COUNT
NEW_NODE_COUNT = NODE_COUNT + 1
vnodes_to_reassign = VNODE_COUNT / NEW_NODE_COUNT
while vnodes_to_reassign > 0:
    for node_to_take_from in xrange(NODE_COUNT):
        for vnode_id, node_id in enumerate(new_vnode2node):
            if node_id == node_to_take_from:
                new_vnode2node[vnode_id] = new_node_id <=========I don't think this is the right mapping
                vnodes_to_reassign -= 1
                if vnodes_to_reassign <= 0:
                    break
        if vnodes_to_reassign <= 0:
            break
moved_ids = 0
for data_id in xrange(DATA_ID_COUNT):
    data_id = str(data_id)
    hsh = unpack_from('>I', md5(str(data_id)).digest())[0]
    vnode_id = bisect_left(vnode_range_starts,
                         hsh % DATA_ID_COUNT) % VNODE_COUNT
    node_id = vnode2node[vnode_id]
    new_node_id = new_vnode2node[vnode_id]
    if node_id != new_node_id:
        moved_ids += 1
percent_moved = 100.0 * moved_ids / DATA_ID_COUNT
print '%d ids moved, %.02f%%' % (moved_ids, percent_moved)

and I put the modified code here:
from bisect import bisect_left
from hashlib import md5
from struct import unpack_from

NODE_COUNT = 100
DATA_ID_COUNT = 10000000
VNODE_COUNT = 1000

vnode_range_starts = []
vnode2node = []
for vnode_id in xrange(VNODE_COUNT):
    vnode_range_starts.append(DATA_ID_COUNT /
                              VNODE_COUNT * vnode_id)
    vnode2node.append(vnode_id % NODE_COUNT)

# I think this is the right method tfo get new_vnode2node
new_vnode2node = []
NEW_NODE_COUNT = NODE_COUNT + 1
for vnode_id in xrange(VNODE_COUNT):
    new_vnode2node.append(vnode_id % NEW_NODE_COUNT)

moved_ids = 0
for data_id in xrange(DATA_ID_COUNT):
    data_id = str(data_id)
    hsh = unpack_from('>I', md5(str(data_id)).digest())[0]
    vnode_id = bisect_left(vnode_range_starts,
                         hsh % DATA_ID_COUNT) % VNODE_COUNT
    node_id = vnode2node[vnode_id]
    new_node_id = new_vnode2node[vnode_id]
    if node_id != new_node_id:
        moved_ids += 1
percent_moved = 100.0 * moved_ids / DATA_ID_COUNT
print '%d ids moved, %.02f%%' % (moved_ids, percent_moved)

and here is the output of the modified code:
8997554 ids moved, 89.98%

I am looking forward to your reply, thanks for your attention.

Best regards,
Ran