Score:0

Unable to start Ceph RGW service

bj flag

Context: I'm trying to setup a new ceph-17.2.5 cluster Initial setup consists of 1 mgr, 1 mon and 1 osd all installed on same box using cephadm.

So far I am able to access http://192.168.3.50:8080/api/auth and get token in the response.

But http://192.168.3.50:8080/api/rgw/bucket returns:

{
    "status": "500 Internal Server Error",
    "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.",
    "request_id": "f08247f9-d9ea-4f88-99b7-d07d5141d5ea"
}

This seems because the dashboard shows rgw service is not running, I have tried to setup rgw service using below commands:

sudo cephadm shell --mount radosgw.yml:/var/lib/ceph/radosgw/radosgw.yml
ceph orch apply -i /var/lib/ceph/radosgw/radosgw.yml

radosgw.yml file contents:

service_type: rgw
service_id: default
placement:
  hosts:
  - <machine-hostname>
  count_per_host: 1
spec:
  rgw_realm: default
  rgw_zone: default
  rgw_frontend_port: 8123
networks:
  -  192.168.3.0/24

After running above the cluster in ceph dashboard does show one rgw service: enter image description here

But the object gateway daemon still gives error as below: enter image description here

The RGW API still returns same error.

Looking at the rgw logs I see rgw keeps trying restart on a loop and eventually stops:

Nov 17 05:31:16 <hostname> systemd[1]: Started Ceph rgw.default.<hostname>.rerude for 76ec21dc-4f10-11ed-8fda-0050568dc50c.
Nov 17 05:31:17 <hostname> bash[2537663]: debug 2022-11-17T05:31:17.012+0000 7fc0de7ed5c0  0 deferred set uid:gid to 167:167 (ceph:ceph)
Nov 17 05:31:17 <hostname> bash[2537663]: debug 2022-11-17T05:31:17.012+0000 7fc0de7ed5c0  0 ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process radosgw, pid 7
Nov 17 05:31:17 <hostname> bash[2537663]: debug 2022-11-17T05:31:17.012+0000 7fc0de7ed5c0  0 framework: beast
Nov 17 05:31:17 <hostname> bash[2537663]: debug 2022-11-17T05:31:17.012+0000 7fc0de7ed5c0  0 framework conf key: endpoint, val: 192.168.3.59:8123
Nov 17 05:31:17 <hostname> bash[2537663]: debug 2022-11-17T05:31:17.012+0000 7fc0de7ed5c0  1 radosgw_Main not setting numa affinity 
Nov 17 05:31:17 <hostname> bash[2537663]: debug 2022-11-17T05:31:17.015+0000 7fc0de7ed5c0  1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0 
Nov 17 05:31:17 <hostname> bash[2537663]: debug 2022-11-17T05:31:17.015+0000 7fc0de7ed5c0  1 D3N datacache enabled: 0 
Nov 17 05:36:17 <hostname> bash[2537663]: debug 2022-11-17T05:36:17.013+0000 7fc0dd36f700 -1 Initialization timeout, failed to initialize 
Nov 17 05:36:17 <hostname> systemd[1]: [email protected].<hostname>.rerude.service: Main process exited, code=exited, status=1/FAILURE 
Nov 17 05:36:17 <hostname> systemd[1]: [email protected].<hostname>.rerude.service: Failed with result 'exit-code'. 
Nov 17 05:36:27 <hostname> systemd[1]: [email protected].<hostname>.rerude.service: Service RestartSec=10s expired, scheduling restart. 
Nov 17 05:36:27 <hostname> systemd[1]: [email protected].<hostname>.rerude.service: Scheduled restart job, restart counter is at 4. 
Nov 17 05:36:27 <hostname> systemd[1]: Stopped Ceph rgw.default.<hostname>.rerude for 76ec21dc-4f10-11ed-8fda-0050568dc50c. 
Nov 17 05:36:27 <hostname> systemd[1]: Started Ceph rgw.default.<hostname>.rerude for 76ec21dc-4f10-11ed-8fda-0050568dc50c. 
Nov 17 05:36:27 <hostname> bash[2538424]: debug 2022-11-17T05:36:27.465+0000 7f2589bf95c0  0 deferred set uid:gid to 167:167 (ceph:ceph) 
Nov 17 05:36:27 <hostname> bash[2538424]: debug 2022-11-17T05:36:27.465+0000 7f2589bf95c0  0 ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process radosgw, pid 6 
Nov 17 05:36:27 <hostname> bash[2538424]: debug 2022-11-17T05:36:27.465+0000 7f2589bf95c0  0 framework: beast 
Nov 17 05:36:27 <hostname> bash[2538424]: debug 2022-11-17T05:36:27.465+0000 7f2589bf95c0  0 framework conf key: endpoint, val: 192.168.3.50:8123 
Nov 17 05:36:27 <hostname> bash[2538424]: debug 2022-11-17T05:36:27.465+0000 7f2589bf95c0  1 radosgw_Main not setting numa affinity 
Nov 17 05:36:27 <hostname> bash[2538424]: debug 2022-11-17T05:36:27.468+0000 7f2589bf95c0  1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0 
Nov 17 05:36:27 <hostname> bash[2538424]: debug 2022-11-17T05:36:27.468+0000 7f2589bf95c0  1 D3N datacache enabled: 0 
Nov 17 05:41:27 <hostname> bash[2538424]: debug 2022-11-17T05:41:27.466+0000 7f258877b700 -1 Initialization timeout, failed to initialize

Not sure what should I look for to get the rgw apis to work in a minimal setup.

us flag
Cross-posting is usually not the best idea. I commented at https://stackoverflow.com/questions/74485739/unable-to-start-ceph-rgw-service
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.