How energy shield users defend themselves against physical damage?

without armour and evasion how casters survive to physical damage late game?

By getting Armour or Evasion like other characters. Or use lightning coil.

For the longest time I ran very tanky characters with lots of defenses but have been recently been playing a low life build with pretty much zero armor or evasion

A shield with good block + rumi's concoction = max block for me

High clear speed keeps my flask often full

Lot's of experience with the game and knowing when to use flasks (granite, Jade, and/or rumi's helps a lot)

Having instant leech is pretty amazing too if you have high enough ES that you are unlikely to get 1 shot by most things.

I haven't really had any problems with phys damage yet.

Playing a low life nado ranger atm and I really dislike ES for this build over life. I have to self cast IC on phys reflect unless I have Vaal disc / Grace up + flasks.

I guess Immortal Call is an alternative. If you get an extra endurance charge or two, that with quality IC + inc duration it can last a while. To make it even better link to enhance. Not everyone wants to cast IC though.

If you got aura nodes, grace is amazing for getting a good amount of evasion from one source. 30-40+% chance to evade or even much more depending on your level and amount buffing it.

i'm a witch with stormcall build

CI: high ES and "Ghost Reaver" (Lifeleech for spells is hard to get though)

Some people say "Lighning Coil" UNique helps as well, but as CI I never tried it.

i'm not going to take CI

While lightning coil is an amazing armor, I would never recommend for any ES based character. The biggest portion of your ES come from your chest and wearing one with zero would be kinda silly in my opinion.

My best suggestion is: if your build can't viably get decent evasion or armor from the tree (it most likely can't unless you run IR and grace or Ondars and Grace) then just proper decison making, positioning and flask management will do wonders.

I feel most non veteran players really underestimate how powerful flasks are if used properly.

You could also try and CWDT - Immortal call setup ... or even manually cast EC and IC if you are doing some boss with tons of phys dmg.

Or just run Iron reflexes with grace.

Energy shield is your defense against physical damage. It has the highest defense against one high dmg hits, both elemental and physical. (Not talking about CI or low-life users)

It just offers you a different play style. Use Ghost Reaver and/or Zealoths Oath. Not using ZO can be a choice for more defense against chaos dmg. I prefer playing ranged style and always having my es and life full and retreat before i start taking dmg on my life pool and let es recharge.

Use cwdt with immortal call+incr duration too avoid getting multiple big hits (many rhoas charging and stunning you, cast on stun also good for this), self cast high lvl molten shell, cwdt summon skelletons/decoy totem ..and high block chance if you can invest in block passives nodes.

Energy shield has consistently been THE highest EHP available. In terms of mitigating big hits, nothing surpasses ES. Its weakness lies in the fact that flasks do not regenerate ES, and ES doesn't regenerate if say you take many successive hits, regardless of big or small.

The best ways to get around this include making use of equipments, skills, and/or passives.
The following are some of the stronger combinations

CwDT + Increased Duration + Immortal Call + Enduring Cry
Spell Totem + Summon Skeletons + Increased Minion Life
Anbu's Charge + Low Life + Zealot's Oath
Springleaf + Low Life + Zealot's Oath
Shavronne's Revelation + Right Ring Slot
Ghost Reaver + Life Leech

A couple of things to consider;
Springleaf has great synergy with Shavronne's Wrappings and Solaris Lorica.
Anbu's Charge is the weakest of these four due to chaos damage bypassing ES and directly hitting life.

My personal favourite build right now involves a hybrid life/es consisting of 5k life, 6k es, 70% chaos resistance, 5% life regen, life leech and Ghost Reaver.