Liver immunology is very specific and immunotolerance is particularly developed due to the constant and massive influx of antigens induced by the proximity of the liver with the digestive tract. Deregulation of liver immunotolerance is involved in the development of chronic liver diseases and can promote hepatic carcinogenesis with complex pathophysiological mechanisms. This aberrant immunotolerance suggests that hepatocellular carcinoma (HCC) is a good candidate for immunotherapies such as immune checkpoint inhibitors, which aim to restore anti-tumor immunity, failing in this situation. However, the objective tumor response rates in HCC patients treated with PD-L1/PD-1 inhibitors are only around 15-20%, suggesting that these therapies work only under certain conditions. In addition, sustained responses can be observed in these responders, suggesting that their identification, thanks to the development of effective predictive response markers, is one of the key challenges for improving the management of HCC. In this short review, we describe the main pathways involved in liver immunotolerance and in HCC immune escape, and we discuss the different prognostic and predictive response markers that could help to optimally select patients with HCC in the future.