This phase I trial tests the safety, side effects, and best dose of eltanexor in combination with venetoclax for the treatment of patients with myelodysplastic syndrome (MDS) or acute myeloid leukemia (AML) that has come back after a period of improvement (relapsed) or that has not responded to previous treatment (refractory). Eltanexor works by trapping "tumor suppressing proteins" within the cell, thus causing the cancer cells to die or stop growing. Venetoclax is in a class of medications called B-cell lymphoma-2 (BCL-2) inhibitors. It may stop the growth of cancer cells by blocking Bcl-2, a protein needed for cancer cell survival. Giving eltanexor together with venetoclax may be safe, tolerable and/or effective in treating patients with relapsed or refractory MDS or AML.