Score:2

Dovecot IMAP Server with Horde Groupware ActiveSync - IMAP Server not reporting changed state

sb flag

I have a Horde Groupware with a Dovecot IMAP Server on Debian Linux running.

Using the Web UI of Horde works well, but I have a problem using ActiveSync.

When using a client app like BlueMail on Android, I do not see the most recent e-mail messages in INBOX.

The activesync log says the following:

[163168][2022-11-11T15:08:44+00:00] >>>: INBOX IMAP PREVIOUS MODSEQ: 190086

This line shows that the LAST time the INBOX was synced, it had a MODSEQ value of 190086. The modseq value reported by the IMAP server must increase anytime there is a message added to the mailbox.

[163168][2022-11-11T15:08:44+00:00] >>>: IMAP status: a:4:{s:8:"messages";i:6874;s:7:"uidnext";i:157933;s:11:"uidvalidity";i:1291740963;s:13:"highestmodseq";i:190086;}

This line shows the CURRENT MODSEQ reported by my IMAP server, showing that the server is not reporting any changed state.

How can that be possible? I got new e-mail in the INBOX, but it's not showing up in the app.

Accessing the IMAP Server directly via Thunderbird, all new e-mail is shown.

It worked before with the Dovecot server on FreeBSD, but I have migrated the Dovecot to Debian Linux.

Any ideas what could be wrong with my Dovecot configuration?

I am using mostly the standard config for Dovecot, especially the IMAP settings.

Here is my imap specific config:

##
## IMAP specific settings
##

# If nothing happens for this long while client is IDLEing, move the connection
# to imap-hibernate process and close the old imap process. This saves memory,
# because connections use very little memory in imap-hibernate process. The
# downside is that recreating the imap process back uses some resources.
#imap_hibernate_timeout = 0

# Maximum IMAP command line length. Some clients generate very long command
# lines with huge mailboxes, so you may need to raise this if you get
# "Too long argument" or "IMAP command line too large" errors often.
#imap_max_line_length = 64k

# IMAP logout format string:
#  %i - total number of bytes read from client
#  %o - total number of bytes sent to client
#  %{fetch_hdr_count} - Number of mails with mail header data sent to client
#  %{fetch_hdr_bytes} - Number of bytes with mail header data sent to client
#  %{fetch_body_count} - Number of mails with mail body data sent to client
#  %{fetch_body_bytes} - Number of bytes with mail body data sent to client
#  %{deleted} - Number of mails where client added \Deleted flag
#  %{expunged} - Number of mails that client expunged, which does not
#                include automatically expunged mails
#  %{autoexpunged} - Number of mails that were automatically expunged after
#                    client disconnected
#  %{trashed} - Number of mails that client copied/moved to the
#               special_use=\Trash mailbox.
#  %{appended} - Number of mails saved during the session
#imap_logout_format = in=%i out=%o deleted=%{deleted} expunged=%{expunged} \
#  trashed=%{trashed} hdr_count=%{fetch_hdr_count} \
#  hdr_bytes=%{fetch_hdr_bytes} body_count=%{fetch_body_count} \
#  body_bytes=%{fetch_body_bytes}

# Override the IMAP CAPABILITY response. If the value begins with '+',
# add the given capabilities on top of the defaults (e.g. +XFOO XBAR).
#imap_capability =

# How long to wait between "OK Still here" notifications when client is
# IDLEing.
#imap_idle_notify_interval = 2 mins

# ID field names and values to send to clients. Using * as the value makes
# Dovecot use the default value. The following fields have default values
# currently: name, version, os, os-version, support-url, support-email,
# revision.
#imap_id_send =

# ID fields sent by client to log. * means everything.
#imap_id_log =

# Workarounds for various client bugs:
#   delay-newmail:
#     Send EXISTS/RECENT new mail notifications only when replying to NOOP
#     and CHECK commands. Some clients ignore them otherwise, for example OSX
#     Mail (<v2.1). Outlook Express breaks more badly though, without this it
#     may show user "Message no longer in server" errors. Note that OE6 still
#     breaks even with this workaround if synchronization is set to
#     "Headers Only".
#   tb-extra-mailbox-sep:
#     Thunderbird gets somehow confused with LAYOUT=fs (mbox and dbox) and
#     adds extra '/' suffixes to mailbox names. This option causes Dovecot to
#     ignore the extra '/' instead of treating it as invalid mailbox name.
#   tb-lsub-flags:
#     Show \Noselect flags for LSUB replies with LAYOUT=fs (e.g. mbox).
#     This makes Thunderbird realize they aren't selectable and show them
#     greyed out, instead of only later giving "not selectable" popup error.
#
# The list is space-separated.
#imap_client_workarounds =

# Host allowed in URLAUTH URLs sent by client. "*" allows all.
#imap_urlauth_host =

# Enable IMAP LITERAL- extension (replaces LITERAL+)
#imap_literal_minus = no

# What happens when FETCH fails due to some internal error:
#   disconnect-immediately:
#     The FETCH is aborted immediately and the IMAP client is disconnected.
#   disconnect-after:
#     The FETCH runs for all the requested mails returning as much data as
#     possible. The client is finally disconnected without a tagged reply.
#   no-after:
#     Same as disconnect-after, but tagged NO reply is sent instead of
#     disconnecting the client. If the client attempts to FETCH the same failed
#     mail more than once, the client is disconnected. This is to avoid clients
#     from going into infinite loops trying to FETCH a broken mail.
#imap_fetch_failure = disconnect-immediately

protocol imap {
  # Space separated list of plugins to load (default is global mail_plugins).
  #mail_plugins = $mail_plugins

  # Maximum number of IMAP connections allowed for a user from each IP address.
  # NOTE: The username is compared case-sensitively.
  #mail_max_userip_connections = 10

An my maildir specific settings:


##
## Maildir-specific settings
##

# By default LIST command returns all entries in maildir beginning with a dot.
# Enabling this option makes Dovecot return only entries which are directories.
# This is done by stat()ing each entry, so it causes more disk I/O.
# (For systems setting struct dirent->d_type, this check is free and it's
# done always regardless of this setting)
#maildir_stat_dirs = no

# When copying a message, do it with hard links whenever possible. This makes
# the performance much better, and it's unlikely to have any side effects.
#maildir_copy_with_hardlinks = yes

# Assume Dovecot is the only MUA accessing Maildir: Scan cur/ directory only
# when its mtime changes unexpectedly or when we can't find the mail otherwise.
maildir_very_dirty_syncs = no

# If enabled, Dovecot doesn't use the S=<size> in the Maildir filenames for
# getting the mail's physical size, except when recalculating Maildir++ quota.
# This can be useful in systems where a lot of the Maildir filenames have a
# broken size. The performance hit for enabling this is very small.
#maildir_broken_filename_sizes = no

# Always move mails from new/ directory to cur/, even when the \Recent flags
# aren't being reset.
#maildir_empty_new = no

Anyone know why MODSEQ is not updated?

dovecot -n:

# 2.3.13 (89f716dc2): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.13 (cdd19fe3)
# OS: Linux 5.10.0-19-amd64 x86_64 Debian 11.5
# Hostname: mail01.public.forberger-online.de
auth_mechanisms = plain login
mail_location = maildir:~/.maildir
mail_privileged_group = mail
mailbox_list_index_include_inbox = yes
namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix =
}
passdb {
  driver = pam
}
protocols = " imap"
service auth {
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0660
    user = postfix
  }
}
service imap-login {
  client_limit = 256
  inet_listener imap {
    address = 192.168.108.1, 127.0.0.1, [::1]
    port = 143
  }
  inet_listener imaps {
    address = 192.168.108.1, 127.0.0.1, [::1]
    port = 993
    ssl = yes
  }
  process_min_avail = 6
}
ssl_ca = </etc/letsencrypt/live/mail.forberger-online.de/chain.pem
ssl_cert = </etc/letsencrypt/live/mail.forberger-online.de/cert.pem
ssl_client_ca_dir = /etc/ssl/certs
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
ssl_min_protocol = TLSv1.2
userdb {
  driver = passwd
}

Important: MODSEQ on folder INBOX does not increase:

* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN AUTH=LOGIN] Dovecot (Debian) ready.
A login REDACTED REDACTED
A OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE] Logged in
A status INBOX (messages)
* STATUS INBOX (MESSAGES 7247)
A OK Status completed (0.001 + 0.000 secs).
1 SELECT Inbox
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft $MDNSent Junk NonJunk $label4 $label1 $label2 $label3 $label5 Old $Forwarded NOTJUNK $has_cal)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $MDNSent Junk NonJunk $label4 $label1 $label2 $label3 $label5 Old $Forwarded NOTJUNK $has_cal \*)] Flags permitted.
* 7247 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1291740963] UIDs valid
* OK [UIDNEXT 158406] Predicted next UID
* OK [HIGHESTMODSEQ 190086] Highest
1 OK [READ-WRITE] Select completed (0.010 + 0.000 + 0.009 secs).

As you can see, HIGHESTMODSEQ stays at 190086. But i received a plenty of new e-mails.

What can be the problem why HIGHESTMODSEQ does not increase?

Ronny Forberger avatar
sb flag
I added the output of dovecot -n. When checking the dovecot log, I see requests coming in from Horde, but I get no new e-mail in AndroidMail. I have only 3 simultaneous clients, Thunderbird, Horde and Prometheus. Any idea what could be wrong?
Ronny Forberger avatar
sb flag
Well, no my file system is on ext4. doveadm index doesn't solve the problem. In all other folders, my e-mails are synced correctly using AndroidMail, but in Outlook for Android I don't get any e-mail in any folder synchronized. I have completely reinstalled my Horde, Nginx and Apache2, but the problem still persists. Well, I have not disabled `lmtp`, I just have not installed the `dovecot-lmtpd` package on Debian. Do I really need it? I thought Postfix (in my case) delivers the mail.
anx avatar
fr flag
anx
I *think* dovecot index not changing things proves that this is not your problem (I thought letting dovecot deliver would ensure that indexes get updated reliably, but if this is not the problem then that is not the solution)
Ronny Forberger avatar
sb flag
OK. I was suposing, my mod_security running in Nginx in front of the Apache2 was the problem, but it is not, I bypassed it completly by exposing Apache2 directly to the Internet. I don't think it's an IMAP problem anymore, rather than I think it's a Horde problem, but the Horde people say, it's an IMAP problem after analyzing my activesync log.
Ronny Forberger avatar
sb flag
But anyway, do you know any way to test the IMAP server if it increases the MODSEQ? I have no problems getting new e-mail with direct IMAP access and Thunderbird though.
Ronny Forberger avatar
sb flag
I found it out myself, how to test it. I updated the question with the IMAP transscript. Can you maybe tell why HIGHESTMODSEQ does not update although I received a plenty of new e-mail?
Score:0
sb flag

It works again now. I recreated the maildir (I only have one) using maildirmake and moved the messages back. Thanks for all help.

Score:0
fr flag
anx

do you know any way to test the IMAP server if it increases the MODSEQ

Seems like the next logical step is to manually connect and

  1. stick to commands that should produce an error when the (expected and advertised) CONDSTORE feature is unavailable.
  2. check which action (receiving a message, moving a message between folders using thunderbird drag'n'drop, executing doveadm index, doveadm force-resync, ..) does and which does not increment the HIGHESTMODSEQ reported by Dovecot, when repeatedly requested with a SELECT before and after tested actions.

Like this:

# openssl s_client -quiet -crlf -connect [::1]:993
* OK [CAPABILITY IMAP4rev1  .. AUTH=LOGIN ..] Dovecot (Debian) ready.
A LOGIN [email protected] password
A OK [CAPABILITY IMAP4rev1  .. CONDSTORE QRESYNC ..] Logged in
A ENABLE CONDSTORE QRESYNC
* OK [HIGHESTMODSEQ 1337] Highest
* ENABLED CONDSTORE QRESYNC
A OK ENABLED (..).
A SELECT INBOX (CONDSTORE)
...
* OK [HIGHESTMODSEQ 1338] Highest
A OK [READ-WRITE] SELECT completed (..).
A SELECT INBOX (CONDSTORE)
...
* OK [HIGHESTMODSEQ 1400] Highest
A OK [READ-WRITE] SELECT completed (..).

After confirming that the problem already exists at the IMAP level, you can continue at the doveadm mailbox status -u [email protected] all NewlyCreatedFolder level. Or even investigating where exactly the number 190086 is coming from. I am thinking of an edge case where Dovecot is incorrectly reporting a number referring to something that cannot be updated because it is not accessible any more. Reading the text representation of the index file via doveadm dump -t index /file/path/to/mailbox/dovecot.index | grep -B 10 -A 100 "modseq.*190086" would then point you to the message - adding an IMAP flag on that message should trigger some result, which might point to the nature of the problem.

Ronny Forberger avatar
sb flag
OK, I did that. The IMAP reports that the `CONDSTORE` feature is available, but the `HIGHESTMODSEQ` stays at 190086, nevertheless which command I execute (Thunderbird drag'n'drop, `doveadm index`, `doveadm force-resync`. I think that is the problem.
anx avatar
fr flag
anx
@RonnyForberger Any progress? I am intrigued, as I was unable to trigger this outside of very clearly logged error conditions. Were you able to you replicate the problem with a *newly created* additional user and mailbox structure? Did you confirm from recent file modification timestamps that Dovecot is not stuck on syncing the index of *a different* (huge? broken?) mailbox and this never managed to finish committing changes on the selected one?
Ronny Forberger avatar
sb flag
I tried what you suggested. When creating a fresh user, with a new maildir created through maildirmake, the problem goes away. I also compared the modification timestamps in my problematic maildir, and I do not get modification of all the meta files when I request the IMAP content with my Groupware. I guess it is a maildir format problem or similiar then. I created the maildir arround year 2002 and copied it several times to different servers and OS'es and but it's not that huge (arround 30000 messages). Is there any migration of the format of the maildir that I maybe missed?
anx avatar
fr flag
anx
@RonnyForberger The format of the mail themselves is fairly static, but a bug in the server or a unsupported migration to an older version might have corrupted log/index files somewhere along the way - depending on what you want to avoid (having clients consider all mail as "new" and download them again, losing flags, ..) you could move & recreate just the metadata while preserving the messages & folders.
Ronny Forberger avatar
sb flag
OK, thanks. See my answered question.
Ronny Forberger avatar
sb flag
By the way, I think it was the downgrade of the dovecot version from FreeBSD to Debian Linux which broke the maildir. I wasn't aware of that.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.