Clock Skew and Proxy Failures: Diagnosing TLS, Tokens, and Signatures
Table of contents
- Fundamentals: why time is part of the protocol, not just a number on screen
- Deep dive: where exactly time is critical
- How the error looks in different clients
- Why clocks drift: anatomy of the drift
- One-minute diagnostics: putting a diagnosis in place quickly
- Configuring synchronization: making it actually work
- Containers and clocks: what's inherited and what isn't
- Common mistakes: what not to do
- Tools and resources
- Cases and results
- Faq: deep answers to frequent questions
Picture this: yesterday your scraper through a proxy worked flawlessly, logs clean, metrics green. This morning you open the dashboard and see a wall of errors: certificate is not yet valid, token expired, signature does not match. The engineer's first thought is predictable: the proxy broke, the provider did something, time to swap the pool. You spend hours on network diagnostics, change endpoints, write to support. And the cause was sitting right under your nose the whole time, ticking wrong. It's the system clock.
Clock skew is one of the most underrated sources of failures in proxy-based infrastructure. It's sneaky because it disguises itself as network and certificate problems. You see the word certificate in the error and reflexively go dig into TLS, even though the certificate is perfectly alive and valid. Your machine just thinks it's a different day.
In this guide, we'll cover the topic from A to Z. You'll learn exactly where time is baked into cryptography and protocols, what concrete errors look like in curl, Python, and Node, why clocks drift in VMs and containers, how to diagnose the issue in a minute, and how to configure synchronization so it actually works rather than just being nominally installed. This material was written by Proxeon engineers based on real incident analysis. One thing to emphasize: this isn't about issuing certificates or how they're built - only about time as a cause of failures.
Fundamentals: why time is part of the protocol, not just a number on screen
Let's start with the foundation. Many people see system time as a purely human convention: it's handy to know it's 2:30 PM. But in the world of network protocols, time is an active participant in security checks. It's baked into validation logic at several levels at once.
When two nodes establish a secure connection or exchange signed messages, they need a way to distinguish fresh data from stale data. Without a concept of time, you can't answer simple questions: has this certificate expired? has this token gone bad? is an attacker replaying an old intercepted request? That's exactly why timestamps and validity windows are built into protocols.
What system time is and where it comes from
Every operating system has two related concepts. First is the hardware clock (RTC, real-time clock), a chip with its own battery power that keeps ticking even when the computer is off. Second is the system clock, which the OS kernel maintains in RAM, starting from the RTC value at boot and adjusting as it runs.
The problem is that the quartz oscillator in any hardware isn't perfect. It runs fast or slow by fractions of a second per day. This is called clock drift. Over a week without correction, noticeable seconds accumulate, and in some virtual environments, whole minutes. To combat drift, the network time synchronization protocol was invented. A synchronization daemon periodically asks reference servers for the exact time and gently nudges the local clock.
UTC, time zones, and why this matters for proxies
The key insight for newcomers: all serious network cryptography operates in UTC - Coordinated Universal Time, with no time zone attachment. Certificates, JWTs, request signatures - everything operates on moments in UTC. Time zone is just cosmetics for human display.
This means that if you've set the time zone wrong but the absolute time (in UTC) is correct, cryptography won't suffer. But if the absolute time itself is off - everything falls apart. A common confusion: an engineer sees a strange local time in the logs, rushes to fix the time zone, and the root cause is elsewhere. Remember the distinction: time zone affects display, absolute time in UTC affects validation.
How proxies fit into this picture
When you work through a proxy, you get an extra node in the request path. But it's important to understand: proxies in most scenarios don't change or substitute time in your cryptographic checks. The TLS handshake with the target server, certificate expiry validation, token validation - all of this happens on your side or on the final server side. The proxy merely passes bytes.
Hence a paradox: working through a proxy doesn't create a time problem, but it makes its symptoms more confusing. The engineer sees the chain of client - proxy - server and naturally suspects the middle link. But the culprit is the local machine where the clock is wrong. We call this the displaced suspicion effect: the longer the chain, the more eagerly we blame its middle rather than its ends.
Deep dive: where exactly time is critical
Now let's go deeper and break down the specific points where wrong time turns into failure. There are four of them, and each deserves separate attention.
Certificate expiry validation in TLS
Every TLS certificate contains two fields: notBefore (not valid before) and notAfter (not valid after). These are the boundaries of the validity window. When your client establishes a secure connection, it receives the server's certificate and checks: does the current time fall within this window?
And here's the key point: the current time means the time on your machine. If your clock is behind and shows a date before notBefore, the client will decide the certificate hasn't started being valid yet. Error like certificate is not yet valid. If the clock has run ahead past notAfter - the certificate has already expired for you, even though for the rest of the world it's fresh. Error certificate has expired.
Short-lived certificates are especially treacherous. Modern practice is moving toward certificates with lifetimes of 90 days and less, and by 2026 the industry is discussing reducing lifetimes to 45 days and below. The shorter the validity window, the less margin against skewed clocks. Previously, an hour of desynchronization was almost invisible against a yearly certificate. Now a narrow window means that even a shift of a few hours at the very edge of renewal can bring down a connection.
JWT: exp, nbf, and iat fields
JSON Web Token is a popular format for authorization tokens. Inside it live time fields that are checked every time the token is used:
- exp (expiration time) - the moment after which the token is considered expired.
- nbf (not before) - the moment before which the token isn't yet valid.
- iat (issued at) - when the token was issued.
All three are Unix timestamps in seconds from epoch, i.e., absolute time in UTC. When the server receives a token, it compares these fields against its own clock. When your client decides whether to refresh a token, it looks at exp relative to its own clock.
The failure scenario is elegantly nasty. Suppose your client's clock has run ten minutes ahead. The server issued a token with a five-minute lifetime. Your client, looking at its fast-running clock, instantly considers a fresh token already expired and either doesn't send it or launches an infinite refresh loop. The reverse situation: if nbf is skewed relative to your clock, you'll get token used before issued or token not yet valid.
Request signatures with timestamp
Many APIs require each request to be signed, and the signature includes a timestamp. A classic example is HMAC signature schemes where the client forms a string from method, path, body, and current timestamp, then signs it with a secret key. The server repeats the computation and compares signatures.
Here time plays a dual role. First, the timestamp is part of the signed string, so the server must use exactly the same timestamp the client sent - it takes it from the header. Second, the server checks that this timestamp isn't too far from its own time. Typically a window of a few minutes is allowed - protection against replay of old requests.
If the client's clock has drifted beyond this window, the server rejects the request as too old or coming from the future. Errors like request timestamp too skewed, signature expired, or the generic signature does not match. And precisely because of time, you often see a signature error rather than a time error - the server doesn't always honestly say the issue is the clock.
One-time codes and time-based passwords
A separate category is time-based one-time passwords (TOTP) used in two-factor authentication when accessing control panels and API consoles. Such a code is computed from a shared secret and current time, broken into intervals of usually 30 seconds. Both sides compute the code independently and compare.
If the client's clock is skewed by more than one or two intervals, the codes stop matching. You enter a freshly generated code, and the system says it's wrong. People in this situation blame the generator app or panic about a breach, when all they had to do was look at the clock. The tolerance here is very narrow - tens of seconds - so TOTP works excellently as an indicator of clock skew.
How the error looks in different clients
Theory is theory, but engineers live in the terminal reading error messages. Let's break down how clock skew manifests in popular tools. This will help you instantly recognize the symptom.
curl
When working with TLS through curl, skewed clocks produce characteristic messages. If time is behind and the certificate hasn't started being valid by your clock:
curl: (60) SSL certificate problem: certificate is not yet validIf time has run ahead and the certificate has already expired by your measure:
curl: (60) SSL certificate problem: certificate has expiredA useful detail: error code 60 refers to certificate validation problems. An inexperienced engineer sees the word certificate and goes to check the certificate itself with a view command, confirms the validity dates are fine, and gets stuck. The answer is that certificate dates are compared against your local time. Check an example through a proxy:
curl -x http://user:pass@gateway.proxeon.net:8080 -v https://api.example.com/statusIf in the -v output you see lines about certificate date validation followed by a validity error - first thing, compare date -u with a reference, don't suspect the gateway.
Python (requests and httpx)
In Python on the standard TLS stack, skewed clocks raise an exception during the handshake:
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.example.com', port=443): Max retries exceeded (Caused by SSLError(SSLCertVerificationError("certificate verify failed: certificate is not yet valid")))Note the tail of the message: certificate is not yet valid. It's the same time symptom. When working with JWT, the picture is different - no TLS errors, but the token validation library throws a specific exception:
jwt.exceptions.ImmatureSignatureError: The token is not yet valid (nbf)or
jwt.exceptions.ExpiredSignatureError: Signature has expiredHere the word signature is misleading - it seems the problem is in the cryptographic signature. In fact, expired points to the exp field and your clock. Quick time check right from code:
import time, datetime; print(datetime.datetime.utcnow(), int(time.time()))Compare the resulting Unix timestamp with the reference - a discrepancy of more than a couple of seconds is already suspicious.
Node.js
In Node, TLS errors come with codes. Clock skew is characterized by:
Error: certificate is not yet valid\ncode: 'CERT_NOT_YET_VALID'and
Error: certificate has expired\ncode: 'CERT_HAS_EXPIRED'The codes CERT_NOT_YET_VALID and CERT_HAS_EXPIRED are direct pointers. If you see the first with a live certificate, your clock is behind. If the second with a definitely fresh certificate - the clock is ahead. When working with JWT libraries in Node, you'll get errors named like TokenExpiredError and NotBeforeError. Quick check:
node -e "console.log(new Date().toISOString(), Math.floor(Date.now()/1000))"Summary table of symptoms
Let's collect the patterns into one mental map:
- certificate is not yet valid / CERT_NOT_YET_VALID - clock is behind.
- certificate has expired / CERT_HAS_EXPIRED with a fresh certificate - clock is ahead.
- token not yet valid / nbf / ImmatureSignature - clock is behind relative to the issuing server.
- token expired / ExpiredSignature right after receiving a token - clock is ahead.
- signature does not match / timestamp too skewed - clock has drifted beyond the server's tolerance window.
- Constantly wrong TOTP code - clock is skewed by tens of seconds or more.
Why clocks drift: anatomy of the drift
Understanding the cause is half the solution. Let's break down why clocks go wrong in modern infrastructure more often than you'd think. This especially applies to servers and worker nodes you run proxy traffic through.
Virtual machines and freezing
A virtual machine doesn't have direct access to the physical quartz. Its sense of time is an abstraction maintained by the hypervisor. Usually things are fine as long as the VM runs continuously. But the moment the hypervisor pauses the machine, the magic begins.
The classic scenario is freezing and snapshots. The hypervisor pauses the VM, say for migration or backup. Inside the guest, time effectively stops. When the machine is unfrozen, its system clock lags by exactly the duration of the pause. If that was a minute - you got a minute shift instantly, all at once. For short-lived tokens and narrow signature windows, this is fatal.
It's even worse with restoration from an old snapshot. The machine comes alive with time from the moment the snapshot was taken - that could be hours or days in the past. TLS will immediately start rejecting certificates as not yet valid. Many cloud platforms provide guest agents that sync time after unfreezing, but they aren't always present and working.
Containers
With containers, the story is subtler. A container has no system clock of its own - it uses the host kernel and, consequently, the host's time. That's the good news: if the host is synchronized, the container sees the right time automatically.
The bad news is in the nuances. First, inside a container you usually can't change the system time - it doesn't have the needed privileges, and rightly so. Second, and importantly, the synchronization daemon is often absent inside the container, and that's fine - the host should be doing the synchronization. The problem arises when the host itself isn't synchronized, and you don't notice because you're used to everything being in sync out of the box on the developer's laptop.
Missing synchronization daemon
The most banal and most frequent cause. On minimal server images, the time synchronization daemon may not be installed or not running. The machine boots, takes time from the RTC, then lives on a drifting quartz without correction. Day after day, the shift accumulates.
Images built manually or cloned are especially dangerous. An engineer configured everything on a reference machine, took an image, rolled it out to a hundred nodes - but the synchronization daemon isn't activated there. A hundred nodes start quietly diverging, each in its own direction. While the shift is small, everything works. A week later, the fastest quartzes climb beyond the tolerance window, and you get floating, irreproducible failures on part of the fleet.
Manual tampering and stuck RTC
Sometimes a human breaks time. Someone manually set the date for a test and forgot to revert it. Someone disabled synchronization because it interfered with a specific experiment. A separate woe is a dead RTC battery on a physical server: after reboot the clock resets to the distant past, and until the first synchronization TLS doesn't work at all.
Double time management
A subtle case that's often forgotten. Sometimes two mechanisms fight over time at once: the hypervisor's guest agent and the synchronization daemon inside the OS. They pull the clock in different directions, and you get time oscillating back and forth. This manifests as intermittent failures that can't be caught. The rule is simple: exactly one mechanism should be responsible for time.
One-minute diagnostics: putting a diagnosis in place quickly
Let's get practical. Your goal is to figure out within sixty seconds whether time is the culprit. Here's the step-by-step framework we use at Proxeon when analyzing incidents.
Step one: look at your UTC time
First, find out what your clock thinks, specifically in UTC, to rule out time zone confusion:
date -uWrite down the value. Now compare with a reference.
Step two: compare with an external source
The most reliable way is to ask a network time server for the time and look at the offset. If chrony is installed:
chronyc trackingIn the output, look for the System time line - it shows the system clock offset from the reference. A value like 0.000030 seconds is the ideal. If systemd-timesyncd:
timedatectl show-timesync --all | grep -i offsetAnother quick trick is a one-shot query to a time server without changing the clock:
chronyd -Q 'server pool.ntp.org iburst'It will print the estimated correction. If its magnitude is large - there's your answer.
Step three: check via the HTTP Date header
An insight that saves a ton of time. Almost any web server returns a Date header in the response with the current time in UTC. Compare it with your clock right through your proxy:
curl -sI -x http://user:pass@gateway.proxeon.net:8080 https://example.com | grep -i ^dateCompare with date -u. If the discrepancy is seconds - all good. If minutes - there's your cause. The beauty of the method is that it requires no installed daemons and works even inside a bare container with nothing but curl.
What spread to consider normal
Practical benchmarks developed from experience:
- Up to 1 second - great, nothing to do. A healthy synchronized system holds fractions of a second.
- 1-5 seconds - acceptable for TLS and most JWTs, but already a zone of attention. Request signatures and TOTP still hold, but the margin is shrinking.
- 5-30 seconds - alarm. TOTP starts failing, narrow signature windows are at risk. Synchronization clearly isn't working properly.
- More than 30 seconds - critical. Signatures fail, short-lived tokens fail, and with large shifts, TLS too. Fix immediately.
- Minutes and hours - catastrophe, usually a consequence of freezing, snapshot, or a dead RTC battery.
Golden rule: if the offset is more than five seconds, time should be considered the prime suspect in all failures of TLS, tokens, and signatures.
Configuring synchronization: making it actually work
Diagnosing isn't enough - you need to fix it and prevent recurrence. Let's break down the two main tools on Linux and, most importantly, how to make sure synchronization is actually active, not just installed. This is the key distinction that gets overlooked.
systemd-timesyncd: the simple option
For most client machines and lightweight nodes, the client built into systemd is sufficient. It does simple time protocol synchronization. Enabling:
timedatectl set-ntp trueChecking that synchronization is actually happening:
timedatectl statusLook for two lines. System clock synchronized: yes means the system considers itself synchronized. NTP service: active means the daemon is running. Both should be positive. If synchronized: no while active - the daemon is running but hasn't been able to reach a server yet, or the server is unreachable.
Details on a specific server:
timedatectl show-timesync --allHere you can see which server it connected to and what offset it got. This command separates genuinely working synchronization from decorative.
chrony: the serious option
For servers where resilience matters, especially for VMs at risk of freezing, chrony is preferable. It handles time jumps more intelligently and converges faster after downtime. Install via the package manager, then start the service. Checking operation - the main command:
chronyc trackingBreaking down the key output lines:
- Reference ID - which source it's bound to. If it's 00000000 or a line about no source selected, there's no synchronization.
- Stratum - the level of distance from the reference clock. It's normal to see a small number.
- System time - the current system clock offset. This is your main indicator.
- Last offset and RMS offset - recent and averaged corrections, showing stability.
List of sources and their state:
chronyc sources -vThe asterisk symbol to the left of a server means it's the one selected as the active source. If no source has a selection mark, the daemon is installed but not synchronizing - a typical trap.
How to distinguish installed synchronization from working
This is the very insight worth reading the section for. The fact that a package is installed and even the fact that a service is running don't guarantee synchronization. The service may be spinning and have no access to time servers - for example, a firewall blocking outbound packets on the needed port, or no internal time server in a closed environment.
The real check consists of three questions. First: is an active source selected? Look at the mark in sources or the Reference ID in tracking. Second: what's the current offset? System time should be in fractions of a second. Third: is it updating? Run the check twice with an interval and make sure the numbers are live, not frozen. If all three answers are positive - synchronization is genuinely working.
Monitoring offset as a metric
The professional approach is not to wait for an incident but to watch the offset continuously. Export the System time value into your monitoring system as a regular metric. Set a warning threshold at, say, two seconds and an alarm at five. Then you'll know about the problem before signatures and tokens start failing. The cost of such monitoring is near zero, and the payoff is huge: one prevented nighttime incident justifies everything.
Containers and clocks: what's inherited and what isn't
Containerization deserves a separate deep dive, because engineers have the most misconceptions here. Let's lay out clearly what a container gets from the host and what it doesn't.
What's inherited: time itself
Key fact: a container shares the host kernel, and therefore the system clock. Inside the container date -u will show exactly the same absolute time as on the host. A container has no separate time counter. This is fundamental. The main takeaway follows: for a container to see the right time, you need to synchronize the host, not try to set up synchronization inside the container.
What's not inherited: time zone
But time display is another matter. The time zone is determined by settings inside the container, usually a zone file and an environment variable. A base image often ships with UTC, which, by the way, is good practice for servers. If inside the container the local time looks different from the host, it's almost always a difference in time zones, not the actual time. Check the absolute in UTC before panicking.
Why you shouldn't run a time daemon in a container
A common newcomer mistake is to stuff a synchronization daemon inside a container. This is wrong for two reasons. First, changing the system time is a privileged operation that affects the entire kernel and therefore all containers on the host and the host itself. By default, a container is prohibited from doing this, and rightly so. Giving it this privilege for synchronization means opening a hole and creating a conflict.
Second, it's simply unnecessary: time already comes from the host. The right architecture is one synchronized host, many containers automatically seeing the correct time. If you have an orchestrator with many nodes, synchronization must be ensured on each host node, not in each pod.
The developer laptop trap
A separate warning about a nasty situation. On the developer's laptop, everything works: containers see the right time because the workstation OS is synchronized out of the box. The engineer builds an image, everything is green. The image goes to a server where the host isn't synchronized - and failures begin there. Lesson: test behavior with skewed time, not just with ideal time. Deliberately shift the time in a test environment and watch how the application reacts.
Checking time in a running container
A quick command to peek inside a running container and verify its absolute time:
docker exec -it my_container date -uIf it matches date -u on the host - all good, look for the problem elsewhere. If the container somehow shows a different absolute time, that's a signal of a non-standard, potentially dangerous configuration worth reviewing immediately.
Common mistakes: what not to do
Experience from incident analysis adds up to a list of rakes people keep stepping on. Let's go through them so you can avoid them.
Mistake one: blaming the proxy by reflex
We started with this and we'll repeat it. The word certificate or signature in an error while working through a proxy automatically breeds suspicion toward the network and gateway. Don't give in. The first action on these errors is to check the time, not to change the endpoint. It takes ten seconds and cuts off the most common hidden cause.
Mistake two: fixing the time zone instead of the time
An engineer sees a strange local time in the logs and changes the time zone. The symptom in the logs changes, but cryptography keeps failing because the real absolute time in UTC is still skewed. Always diagnose via date -u and comparison with a reference, not by local display.
Mistake three: assuming installing synchronization is the solution
Installed the package, saw the service running, closed the ticket. A week later failures again because the service had no access to time servers. Installing doesn't equal synchronizing. Always check the actual offset and presence of a selected source.
Mistake four: hard-setting the clock on the fly
A sharp jump in system time via a direct set command can break running processes that rely on time monotonicity: timeouts expire, sessions break, incorrect scheduler triggers fire. The right way is to let the synchronization daemon nudge the clock smoothly. A hard correction is acceptable only with a huge one-time shift, and even then consciously.
Mistake five: ignoring VM freezing
The team doesn't account for migrations, snapshots, and suspensions creating instantaneous shifts. Such environments need a daemon resilient to jumps and offset monitoring after maintenance operations. If your failures correlate in time with backups or migrations - there's your answer.
Mistake six: double time management
The hypervisor guest agent and the internal daemon run simultaneously. The clock jerks, failures are intermittent and don't reproduce. Pick one mechanism and disable the other. This cures the most exhausting floating bugs.
Mistake seven: windows too narrow with no margin
If you're building an API with request signatures, don't set a 30-second tolerance window without a compelling reason. A reasonable margin of a few minutes dramatically reduces sensitivity to minor client clock skew without sacrificing replay protection. The balance between strictness and resilience is an engineering decision, not dogma.
Tools and resources
Let's assemble an arsenal worth keeping handy. All tools are standard and legitimate, for normal engineering operations.
Command line
- date -u - an instant look at absolute time. The first command on any suspicion.
- timedatectl - synchronization and time zone status on systems with systemd.
- chronyc tracking and chronyc sources - deep chrony diagnostics: offset, sources, stability.
- curl -sI ... | grep -i date - checking time via the HTTP response header, works even where there are no daemons. Ideal for bare containers and checks through a gateway.
Language checks
- In Python: a one-liner printing utcnow and timestamp for comparison right from the app's runtime.
- In Node: a one-liner with toISOString and Date.now to see time through the eyes of your specific runtime.
- Decoding a JWT without signature verification to see the exp, nbf, iat fields with your own eyes and match them against the current time. This removes guesswork: you literally see whether the token has expired by your clock or not.
What to monitor continuously
- System clock offset as a numeric metric with warning and alarm thresholds.
- Status of having a selected time source - a boolean indicator of synchronization health.
- Frequency of TLS and token errors broken down by node - a spike on a specific node often points precisely to its skewed clock.
Proxeon infrastructure
When working through Proxeon gateways, we recommend embedding a time check into your worker nodes' startup script. One line checking the Date header through the gateway on startup - and you'll catch desynchronization before the first work request. It's cheap and radically reduces the share of false support tickets where the root turns out to be the client-side clock, not the proxy.
Cases and results
Theory comes alive in real stories. Here are generalized cases from practice - numbers rounded, details anonymized, but the patterns are absolutely real.
Case one: nocturnal scraper collapse after backup
The team collected data through a proxy around the clock. Every night around three o'clock a wall of certificate is not yet valid errors began, and by morning everything self-healed. Engineers spent two weeks blaming the proxy pool, changing endpoints, filing complaints. The answer came when someone noticed the correlation: failures began exactly during the nighttime VM backup.
The hypervisor froze the VM for a minute and a half to two minutes to take a consistent snapshot. After unfreezing, the clock lagged by those minutes, and the guest agent synced them not immediately. In the window between unfreezing and correction, TLS rejected fresh certificates as not yet valid - because by the lagging clock they started being valid in the future. The solution was migrating to chrony with fast convergence after a jump and offset monitoring right after backup operations. Nighttime failures disappeared completely, and the diagnostic time for future similar problems dropped from days to minutes.
Case two: a hundred nodes diverging apart
An organization rolled out a fleet of a hundred worker nodes from one image. The first week everything worked. Then floating request signature failures began on random nodes - request timestamp too skewed. Irreproducible: restart the task, and it might pass on a different node.
The cause was that time synchronization wasn't activated in the image. A hundred nodes drifted, each by its own quartz. The fastest ones went beyond the signature tolerance window within a week. A mass check of chronyc tracking across the entire fleet showed offsets from fractions of a second to fifteen seconds. After enabling and verifying actual synchronization on all nodes, plus adding the offset metric to monitoring, the failures stopped. The team's takeaway: in mass rollouts, check not the fact of installation but the fact of synchronization.
Case three: the developer nobody understood
One engineer complained that locally his TOTP authorization to the control panel didn't work, though it worked for everyone else. He enters the right code, the system rejects it. They suspected issues with his account.
It turned out that a week earlier he had manually shifted the system time on his workstation for testing another app and forgot to revert it, while disabling synchronization. The clock drifted almost a minute. TOTP with a 30-second window stopped matching. Enabling automatic synchronization instantly fixed everything. The moral: the narrow TOTP tolerance is a built-in clock skew detector. If codes don't match, first thing, look at the clock.
Case four: a container with the wrong time zone
In the app logs in the container, time ran with a three-hour shift relative to the host. Engineers decided the container's clock was skewed and spent a day trying to set up synchronization inside it, nearly granting the container extra privileges.
Checking date -u inside and outside showed the same absolute time. The shift was purely in display: the base image had one time zone, the host another. Cryptography worked flawlessly because everything matched in UTC. There was no real problem at all - just cosmetic confusion in the logs. The lesson cost a day of work: always distinguish absolute time from its display.
FAQ: deep answers to frequent questions
Can a proxy itself skew my time or substitute it in TLS?
In standard scenarios working through a gateway - no. The proxy passes bytes between you and the target server. Certificate expiry validation, exp and nbf fields, signature window all happen on your side or on the final server side, based on their own clocks. So on errors resembling time issues, you should suspect the local clock of the worker node first, not the gateway. The proxy merely lengthens the chain and psychologically shifts suspicion toward the middle.
How accurate does the clock need to be for everything to work?
For TLS, the margin is usually large - certificate validity windows are measured in days, and even a minute shift mostly goes unnoticed, except at the very edge of renewal. For JWT, everything depends on token lifetime: with short tokens, even seconds matter. For request signatures, the typical window is minutes, but it's better to keep the offset within a second. TOTP is the most demanding - tens of seconds. Universal recommendation: keep the offset within one second, then you're covered against all the mechanisms listed at once.
Why does the certificate show valid dates, but the client says it's not yet valid?
Because the client compares the certificate dates not with absolute truth but with your local clock. If your clock is behind and shows a moment before the notBefore field, for the client the certificate hasn't started yet. The dates in the certificate itself are perfect. The answer is always in comparing your date -u with a reference. This is the most common source of engineer stupefaction.
Should I install a synchronization daemon inside a container?
No. The container uses the host kernel's clock, so you need to synchronize the host. Installing a daemon inside a container is useless and requires dangerous privileges to change the system time, affecting the whole host. The right model is one synchronized host and many containers automatically seeing the correct time. In an orchestrator, synchronization is ensured on each host node.
How do I distinguish a time problem from a real proxy or network problem?
Start with a time check - it takes ten seconds. Compare date -u with a reference and with the HTTP Date header through your gateway. If the offset is small but TLS and token errors remain - then move on to network diagnostics. If the offset is large - you've found the cause. The key sign of time problems: errors containing the words not yet valid, expired, skewed, signature with definitely live certificates and fresh tokens.
What should I do right after unfreezing or migrating a VM?
Make sure the synchronization daemon quickly synced the clock. For environments with freezing, chrony is preferable, as it handles jumps well. Check chronyc tracking and make sure System time has returned to fractions of a second. Good practice is to forcibly trigger an offset check after maintenance operations and not fire critical signed requests until the clock has converged.
Does a wrong time zone affect TLS and token operation?
No, provided the absolute time in UTC is correct. All cryptography operates in UTC, and the time zone is only display for humans. A wrong time zone will confuse you in the logs but won't bring down TLS, JWT, or signature. That's exactly why you should diagnose via UTC, not via local time. Confusing time zone with absolute time is a classic trap.
How do I embed a time check into a workflow through a proxy?
Add one comparison line to the node's startup script: request the Date header through your Proxeon gateway and compare with local date -u. If the discrepancy exceeds the threshold, halt the startup and raise an alarm. Plus export the system clock offset to monitoring as a permanent metric with thresholds. These two measures catch the vast majority of time incidents before they turn into TLS and token failures.
Why do I get
