Score:0

postgresql stopps working shortly after install

in flag

I have a ubuntu 20.04 server (root server at hetzner.com if that helps anyone) and I am trying to get a simple postgres running. Installation (following this guide https://tecadmin.net/how-to-install-postgresql-in-ubuntu-20-04/) works fine and postgres log looks like:

2021-06-22 08:46:03.902 CEST [2840] LOG: starting PostgreSQL 13.3 (Ubuntu 13.3-1.pgdg20.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, 64-bit
2021-06-22 08:46:03.902 CEST [2840] LOG: listening on IPv4 address "127.0.0.1", port 5432
2021-06-22 08:46:03.908 CEST [2840] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-06-22 08:46:03.926 CEST [2841] LOG: database system was shut down at 2021-06-22 08:46:02 CEST
2021-06-22 08:46:03.935 CEST [2840] LOG: database system is ready to accept connections
2021-06-22 09:03:26.552 CEST [3080] postgres@odc FATAL: password authentication failed for user "postgres"
2021-06-22 09:03:26.552 CEST [3080] postgres@odc DETAIL: User "postgres" has no password assigned.
Connection matched pg_hba.conf line 96: "host all all 127.0.0.1/32 md5"
2021-06-22 09:03:26.554 CEST [3081] postgres@odc FATAL: password authentication failed for user "postgres"
2021-06-22 09:03:26.554 CEST [3081] postgres@odc DETAIL: User "postgres" has no password assigned.
Connection matched pg_hba.conf line 96: "host all all 127.0.0.1/32 md5"
2021-06-22 09:06:58.887 CEST [2840] LOG: received fast shutdown request
2021-06-22 09:06:58.921 CEST [2840] LOG: aborting any active transactions
2021-06-22 09:06:58.923 CEST [2840] LOG: background worker "logical replication launcher" (PID 2847) exited with exit code 1
2021-06-22 09:06:58.923 CEST [2842] LOG: shutting down
2021-06-22 09:06:59.001 CEST [2840] LOG: database system is shut down
2021-06-22 09:06:59.169 CEST [3133] LOG: starting PostgreSQL 13.3 (Ubuntu 13.3-1.pgdg20.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, 64-bit
2021-06-22 09:06:59.169 CEST [3133] LOG: listening on IPv4 address "127.0.0.1", port 5432
2021-06-22 09:06:59.175 CEST [3133] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-06-22 09:06:59.189 CEST [3139] LOG: database system was shut down at 2021-06-22 09:06:58 CEST
2021-06-22 09:06:59.198 CEST [3133] LOG: database system is ready to accept connections
2021-06-22 09:07:16.130 CEST [3181] postgres@odc FATAL: database "odc" does not exist

Note that i made a mistake in psql tryxing to acces a DB that did not exist in last lie od ölog.

However after some time everything stops working ant the log now has the following lines (nothing in between this and the previous block)

2021-06-22 09:50:50.116 CEST [4144] FATAL: could not open file "global/pg_filenode.map": No such file or directory
2021-06-22 09:51:10.111 CEST [3133] LOG: could not open file "postmaster.pid": No such file or directory
2021-06-22 09:51:10.111 CEST [3133] LOG: performing immediate shutdown because data directory lock file is invalid
2021-06-22 09:51:10.112 CEST [3133] LOG: received immediate shutdown request
2021-06-22 09:51:10.112 CEST [3133] LOG: could not open file "postmaster.pid": No such file or directory
2021-06-22 09:51:10.112 CEST [3144] LOG: could not open temporary statistics file "pg_stat/global.tmp": No such file or directory
2021-06-22 09:51:10.112 CEST [3143] WARNING: terminating connection because of crash of another server process
2021-06-22 09:51:10.112 CEST [3143] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2021-06-22 09:51:10.112 CEST [3143] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2021-06-22 09:51:10.112 CEST [4147] WARNING: terminating connection because of crash of another server process
2021-06-22 09:51:10.112 CEST [4147] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2021-06-22 09:51:10.112 CEST [4147] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2021-06-22 09:51:10.116 CEST [3133] LOG: database system is shut down

the folder /var/lib/postgresql/ is missing the main subfolder and ps aux | grep post returns:

postgres 4010 0.0 0.0 200 20 ? Ss 09:50 0:00 iihetsbssbatshhn
postgres 4012 0.0 0.0 0 0 ? Z 09:50 0:00 [hrrscshsbhhitia] <defunct>
postgres 4013 0.0 0.0 0 0 ? Z 09:50 0:00 [hrrscshsbhhitia] <defunct>
postgres 4014 0.0 0.0 0 0 ? Z 09:50 0:00 [hrrscshsbhhitia] <defunct>
postgres 4264 9.5 0.0 0 0 ? Z 10:14 0:58 [hrrscshsbhhitia] <defunct>
postgres 4296 39.9 0.0 264 28 ? S 10:20 1:37 iihetsbssbatshhn
postgres 4297 0.0 0.0 200 28 ? S 10:20 0:00 iihetsbssbatshhn
postgres 4303 0.0 0.0 0 0 ? Z 10:21 0:00 [hrrscshsbhhitia] <defunct>

and sudo systemctl status 'postgresql*' returns:

- [email protected] - PostgreSQL Cluster 13-main
Loaded: loaded (/lib/systemd/system/[email protected]; enabled-runtime; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2021-06-22 09:51:10 CEST; 34min ago
Process: 3121 ExecStart=/usr/bin/pg_ctlcluster --skip-systemctl-redirect 13-main start (code=exited, status=0/SUCCESS)
Process: 4148 ExecStop=/usr/bin/pg_ctlcluster --skip-systemctl-redirect -m fast 13-main stop (code=exited, status=1/FAILURE)
Main PID: 3133 (code=exited, status=0/SUCCESS)
Jun 22 09:06:59 Ubuntu-2004-focal-64-minimal systemd[1]: Starting PostgreSQL Cluster 13-main...
Jun 22 09:07:01 Ubuntu-2004-focal-64-minimal systemd[1]: Started PostgreSQL Cluster 13-main.
Jun 22 09:51:10 Ubuntu-2004-focal-64-minimal postgresql@13-main[4148]: Error: /var/lib/postgresql/13/main is not accessible or does not exist
Jun 22 09:51:10 Ubuntu-2004-focal-64-minimal systemd[1]: [email protected]: Control process exited, code=exited, status=1/FAILURE
Jun 22 09:51:10 Ubuntu-2004-focal-64-minimal systemd[1]: [email protected]: Failed with result 'exit-code'.

- postgresql.service - PostgreSQL RDBMS
Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
Active: active (exited) since Tue 2021-06-22 09:07:01 CEST; 1h 18min ago
Process: 3156 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
Main PID: 3156 (code=exited, status=0/SUCCESS)
Jun 22 09:07:01 Ubuntu-2004-focal-64-minimal systemd[1]: Starting PostgreSQL RDBMS...
Jun 22 09:07:01 Ubuntu-2004-focal-64-minimal systemd[1]: Finished PostgreSQL RDBMS.

I have tried purging and reinstalling postgresql twice. Every time I get the exact same behavior.

Any help is greatly appreciated. Thanks

----- EDIT ----

I have now used postgresql-12 and it seems to have solved the problem. I would still like to find out what the problem was

Nikita Kipriyanov avatar
za flag
Out of file descriptors? Out of memory (what `dmesg` says)? Unclean file system? Not enough inodes? Not enough disk space? More than one Postgres process was started using the same data directory? Something else is doing nasty things with Postgres's directory?
bwright avatar
in flag
the only non ufw message in dmesg is perf: interrupt took too long (2509 > 2500), lowering kernel.perf_event_max_sample_rate to 79500. How would I go about checking for an unclean file system? df -i returns that there are only about 1% inodes used per partition. same for disk space. I would not know what else should do something nasty to the postgres directory (how would i find out).
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.