It's not really the size of the contents of the directory, but it's rather how much space on disk is used to store the meta data(e.g. about it and it's immediate contents inode numbers and names/paths) of that directory.
To simplify it, think of a directory similar to a Telephone directory ... i.e. the more names, numbers and addresses it contains, the larger in size it becomes and vice versa.
Please see the demonstration below:
# Block size
$ stat -fc %s .
4096
# Create empty directories
$ mkdir 1 2 3
$ ls -lh
total 12K
drwxrwxr-x 2 ubuntu ubuntu 4.0K Mar 8 11:43 1
drwxrwxr-x 2 ubuntu ubuntu 4.0K Mar 8 11:43 2
drwxrwxr-x 2 ubuntu ubuntu 4.0K Mar 8 11:43 3
# Create multiple empty 0 size files
$ touch 1/file{1..1000}
$ touch 2/file{1..100000}
# Create a 1G size file
$ fallocate -l 1G 3/file1
$ ls -lh
total 2.7M
drwxrwxr-x 2 ubuntu ubuntu 28K Mar 8 11:44 1
drwxrwxr-x 2 ubuntu ubuntu 2.6M Mar 8 11:44 2
drwxrwxr-x 2 ubuntu ubuntu 4.0K Mar 8 11:44 3
$ du -sh *
28K 1
2.6M 2
1.1G 3
# Copy a more populated directory with larger meta-info size under another less populated directory
$ cp -r 2/ 3/
$ ls -lh
total 2.7M
drwxrwxr-x 2 ubuntu ubuntu 28K Mar 8 11:44 1
drwxrwxr-x 2 ubuntu ubuntu 2.6M Mar 8 11:44 2
drwxrwxr-x 2 ubuntu ubuntu 4.0K Mar 8 11:44 3
$ ls -lh 3/
total 1.1G
drwxrwxr-x 2 ubuntu ubuntu 2.6M Mar 8 11:44 2
-rw-rw-r-- 1 ubuntu ubuntu 1.0G Mar 8 11:44 file1
Notice that the minimum allocatable size for directory metadata/information depends on the file system ... e.g. on Ubuntu EXT4, it defaults to the block size(find it with e.g. stat -fc %s .) ... Other filesystems, however, might utilize less/more than that ... e.g. on NTFS:
# Block size
$ stat -fc %s .
4096
# Create an empty directory
$ mkdir dir_on_ntfs
$ ls -lh
total 0
drwxrwxrwx 1 ubuntu ubuntu 0 Mar 8 13:32 dir_on_ntfs
# Populate the directory with multiple empty 0 size files
$ touch dir_on_ntfs/file{1..10000}
$ ls -lh
total 2.0M
drwxrwxrwx 1 ubuntu ubuntu 2.0M Mar 8 13:32 dir_on_ntfs
And on XFS:
# Block size
$ stat -fc %s .
4096
# Create an empty directory
$ mkdir dir_on_xfs
$ ls -lh
total 0
drwxr-xr-x 2 ubuntu ubuntu 6 Mar 8 13:32 dir_on_xfs
# Populate the directory with multiple empty 0 size files
$ touch dir_on_xfs/file{1..10000}
$ ls -lh
total 376K
drwxr-xr-x 2 ubuntu ubuntu 240K Mar 8 13:32 dir_on_xfs
While on VFAT:
# Block size
$ stat -fc %s .
32768
# Create an empty directory
$ mkdir dir_on_vfat
$ ls -lh
total 32K
drwxr-xr-x 2 ubuntu ubuntu 32K Mar 8 13:32 dir_on_vfat
# Populate the directory with multiple empty 0 size files
$ touch dir_on_vfat/file{1..10000}
$ ls -lh
total 640K
drwxr-xr-x 2 ubuntu ubuntu 640K Mar 8 13:32 dir_on_vfat
