I’ve been encountering a very curious issue with a Podman container, and I’ve been trying to resolve it for days now. I want to run Container (1) with a web server (Apache2/httpd) on a VM. I forward port 80 of this web server to port 8082 on the host/VM, and on the host, there is another web server acting as a reverse proxy. The website within the container should be accessible at www.example.com/a_site/
.
On the same VM, there is another service running with a different web server (Python, Uvicorn) in Container (2), which is forwarded to port 8081 and accessible at www.example.com/another_site/
. This setup works fine.
However, for the web server in Container (1), it’s not working. I get a 403 Forbidden error.
Image Details for the Web Service (Testing Purpose)
The image for testing is very simple:
- It serves a single HTML file that loads an image.png (which is also stored inside the podman image).
- I’ve used the standard web server configuration.
- No PHP, JavaScript, or any other complex features are used.
What I’ve Tried:
- The image for Container (1) works perfectly on my personal computer (different OS (Ubuntu), different Podman version 4.9.3).
- The same image for Container (1) runs on a second VM (our dev VM) with the same setup as the first VM, also without issues. Both VMs use the same OS (Oracle Linux Server 9.6) and release, but the first VM has no internet connection, which hasn’t affected the issue (I tested it).
- I attempted to build the image from a plain
debian-slim
image, installing Apache2 and also starting with anhttpd
image directly. - I tested using Nginx instead of Apache2/httpd inside Container (1), but I encounter the same 403 Forbidden issue. This setup works fine on my personal computer and the second VM.
- I also tested an image with Python’s built-in
http.server
with the same files, and this works fine with no issues. This suggests that the reverse proxy on the host/VM is configured correctly. This image also works on my computer and the second VM.
How I Tested the Issue:
- I can log into the container (1), so the container seems to have started correctly.
- When I try to request the site from outside the container (
curl -I http://localhost:8082
), I get the 403 Forbidden error. - When I try to request the site from inside the container (
curl -I http://localhost:80
), I get the same 403 Forbidden error.
This leads me to believe that the web server inside the container is not running correctly. When I check the Apache error logs (/var/log/apache2/error.log
), I see the following:
[Thu Oct 09 09:34:25.451475 2025] [mpm_event:notice] [pid 17:tid 17] AH00489: Apache/2.4.65 (Debian) configured -- resuming normal operations [Thu Oct 09 09:34:25.451785 2025] [core:notice] [pid 17:tid 17] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND' libgcc_s.so.1 must be installed for pthread_exit to work libgcc_s.so.1 must be installed for pthread_exit to work [Thu Oct 09 09:34:26.453339 2025] [core:notice] [pid 17:tid 17] AH00051: child pid 18 exit signal Abort (6), possible coredump in /etc/apache2 [Thu Oct 09 09:34:26.454774 2025] [core:notice] [pid 17:tid 17] AH00051: child pid 19 exit signal Abort (6), possible coredump in /etc/apache2 libgcc_s.so.1 must be installed for pthread_exit to work libgcc_s.so.1 must be installed for pthread_exit to work ... (continued many times)
I tried fixing this error by installing GCC (apt-get install gcc
) within the image, but it didn’t change anything. Perhaps GCC is not the correct package to install the missing library.
Another Error Seen in the Logs:
Sometimes, I also see this error in the logs:
(13)Permission denied: [client 134.130.30.100:33628] AH00035: access to /image.png denied (filesystem path '/usr') because search permissions are missing on a component of the path
This error and a search for 403 Forbidden suggest that file or folder permissions are missing. I’ve tested changing the permissions and ownership of the files within the container (1) for both the Apache2 and Nginx web servers. The files (e.g., index.html
) are set to 644
, and the containing folder to 755
. This didn’t help. I also tried granting 755
permissions to all parent folders.
However, keep in mind that the same image works on my notebook and the second VM without needing to change any permissions.
Other Possible Causes:
- Server Configration: Even thought I used the standard configuration, I checked all configuration files and I couldn't find any problems.
- AppArmor, SELinux: Neither of these exist within my container, and both are deactivated on the hosts/VMs.
- Firewall: The firewall on the host has been deactivated, and the issue persists.
Conclusion:
All of this suggests that the issue arises during the initialization process of the container, but only on one VM. It seems to affect containers with Apache2/httpd or Nginx web servers, but not the Python web server. This is really frustrating because this is exactly what containers are supposed to solve—running consistently across different environments.
Has anyone encountered this issue or have any ideas on what I should try next? If you need more details, please let me know.